Information processing apparatus, information processing system, and information processing method

Abstract

The information processing device includes sensors, a communication unit, and a circuit configuration. The sensors are configured to acquire relative position information of the main display device relative to the sub-display device. The communication unit is configured to transmit user viewpoint information from the main display device to the sub-display device and / or receive user viewpoint information from the sub-display device. The circuit configuration is configured to: acquire viewpoint information and relative position information; acquire absolute device position information based on the relative position information, indicating the position information of the main display device and the sub-display device in a coordinate system based on the main display device; and control the display of the main display device based on the integrated information based on the viewpoint information and the absolute device position information.

Description

Technical Field

[0001] This disclosure relates to information processing equipment, information processing systems, and information processing methods. Background Technology

[0002] In related technologies, spatial display devices are known as a type of glasses-free 3D display device.

[0003] The spatial display device detects the viewpoint position of the user (viewer) using a camera mounted on the device, generates a viewpoint video corresponding to that viewpoint position, and performs panel display. The panel is equipped with light-distributing components such as lenticular lenses, and the video displayed on the panel is directionally separated relative to the user's left and right eyes. This allows the user to view stereoscopic video with the naked eye.

[0004] In recent years, multiple such spatial display devices have been arranged side by side to collaboratively display wide-area scenes, and technologies for achieving this have been proposed (e.g., see PTL 1).

[0005] Reference List

[0006] Patent documents

[0007] PTL 1: WO 2022 / 230350 Summary of the Invention

[0008] Technical issues

[0009] However, the aforementioned prior art has room for further improvement in achieving collaborative control, including at least collaborative display, through a simple combination of multiple spatial display devices. For example, in the aforementioned prior art, in addition to the spatial display devices, position detection also requires a large-scale backend such as imaging equipment and signal processing equipment.

[0010] In response, this disclosure proposes information processing equipment, information processing system and information processing method, which realize collaborative control including at least collaborative display through a simple plurality of spatial display devices.

[0011] Solution to the problem

[0012] To address the aforementioned problems, an information processing apparatus according to one aspect of this disclosure includes a sensor, a communication unit, and a circuit system. The sensor is configured to acquire relative position information of a main display device relative to a sub-display device. The communication unit is configured to perform at least one of the following: transmitting user viewpoint information from the main display device to the sub-display device, or receiving user viewpoint information from the sub-display device. The circuit system is configured to acquire viewpoint information and relative position information, acquire device absolute position information based on the relative position information indicating the position information of the main display device and the sub-display device in a coordinate system based on the main display device, and control the display of the main display device based on integrated information based on the viewpoint information and the device absolute position information. Attached Figure Description

[0013] [ Figure 1 ] Figure 1 This is a diagram illustrating an example configuration of a display system according to an embodiment of the present disclosure.

[0014] [ Figure 2 ] Figure 2 This is a diagram illustrating an example configuration of a spatial display device according to an embodiment of the present disclosure.

[0015] [ Figure 3 ] Figure 3 This is a diagram illustrating how the display system is used.

[0016] [ Figure 4 ] Figure 4 This is a diagram illustrating an example of the intended display of the display system.

[0017] [ Figure 5 ] Figure 5 This is an illustration of a typical human field of vision.

[0018] [ Figure 6 ] Figure 6 This diagram illustrates situations where viewpoint acquisition is difficult in some spatial display devices.

[0019] [ Figure 7 ] Figure 7 This is a diagram showing an example of a partially incorrect display.

[0020] [ Figure 8 ] Figure 8 This is a schematic diagram illustrating an information processing method according to an embodiment of the present disclosure.

[0021] [ Figure 9 ] Figure 9 This is a block diagram illustrating an example configuration of a spatial display device according to an embodiment of the present disclosure.

[0022] [ Figure 10] Figure 10 This is a diagram showing the data flow when the target device is the main display device.

[0023] [ Figure 11 ] Figure 11 This is a diagram showing the data flow when the target device is a sub-display device.

[0024] [ Figure 12 ] Figure 12 It displays the state transition diagram of the system.

[0025] [ Figure 13 ] Figure 13 This is an illustration of the basic structure of a data packet (part 1).

[0026] [ Figure 14 ] Figure 14 This is an illustration of the basic structure of a data packet (part 2).

[0027] [ Figure 15 ] Figure 15 This is a flowchart illustrating the processing procedure when the device is in the UNDER_CONNECTION state.

[0028] [ Figure 16 ] Figure 16 Is with Figure 15 The corresponding processing sequence diagram.

[0029] [ Figure 17 ] Figure 17 This is a flowchart (part 1) showing the processing procedure for the master device in the topology established (TOPOLOGY_ESTABLISHED) state.

[0030] [ Figure 18 ] Figure 18 This is a flowchart (part 2) showing the processing procedure of the master device when the topology has been established.

[0031] [ Figure 19 ] Figure 19 This is a flowchart (part 3) showing the processing procedure of the master device when the topology has been established.

[0032] [ Figure 20 ] Figure 20 This is a flowchart (part 4) showing the processing procedure of the master device in the state where the topology has been established.

[0033] [ Figure 21 ] Figure 21 This is a flowchart (part 5) showing the processing procedure of the master device when the topology has been established.

[0034] [ Figure 22 ] Figure 22 Is with Figures 17 to 21 The corresponding processing sequence diagram.

[0035] [ Figure 23 ] Figure 23 This is a flowchart (part 1) showing the processing procedure for a sub-device in a topology-established state.

[0036] [ Figure 24 ] Figure 24 This is a flowchart (part 2) showing the processing procedure for a sub-device when the topology has been established.

[0037] [ Figure 25 ] Figure 25 Is with Figure 23 and Figure 24 The corresponding processing sequence diagram.

[0038] [ Figure 26 ] Figure 26 This is a diagram (part 1) illustrating the data structure of a display system according to an embodiment of the present disclosure.

[0039] [ Figure 27 ] Figure 27 This is a diagram (part 2) illustrating the data structure of a display system according to an embodiment of the present disclosure.

[0040] [ Figure 28 ] Figure 28 This is a diagram (part 3) illustrating the data structure of a display system according to an embodiment of the present disclosure.

[0041] [ Figure 29 ] Figure 29 This is a diagram (part 4) illustrating the data structure of a display system according to an embodiment of the present disclosure.

[0042] [ Figure 30 ] Figure 30 This is a diagram showing an example of equipment layout.

[0043] [ Figure 31 ] Figure 31 This is an illustration of relative position detection using UWB.

[0044] [ Figure 32 ] Figure 32 This is a diagram showing an example of the UWB detection range.

[0045] [ Figure 33 ] Figure 33 This is a diagram showing the detection results of the local geometric information of device 01.

[0046] [ Figure 34 ] Figure 34This is a diagram showing the detection results of the local geometric information of device 02.

[0047] [ Figure 35 ] Figure 35 This is a diagram showing the detection results of the local geometric information of device 03.

[0048] [ Figure 36 ] Figure 36 This is a diagram showing the detection results of the local geometric information of device 06.

[0049] [ Figure 37 ] Figure 37 This is a diagram showing the detection results of the local geometric information of device 07.

[0050] [ Figure 38 ] Figure 38 It is shown Figures 33 to 37 The detection results are shown in the graph, along with the relative position information.

[0051] [ Figure 39 ] Figure 39 This is a graph showing the relationship between the relative position of the device and the detection results of local geometric information.

[0052] [ Figure 40 ] Figure 40 It is a diagram showing the structure of a relative position diagram.

[0053] [ Figure 41 ] Figure 41 It is a graph that illustrates the data structure of the graph structure.

[0054] [ Figure 42 ] Figure 42 This is a diagram illustrating an example of how graph data is stored.

[0055] [ Figure 43 ] Figure 43 This is a diagram showing the data structure of devPosRotEntry, which illustrates the relative position information between devices.

[0056] [ Figure 44 ] Figure 44 This is a graph showing the initial branches determined by the connection topology.

[0057] [ Figure 45 ] Figure 45 This is a diagram illustrating an example of the additional conditions during connection topology determination.

[0058] [ Figure 46 ] Figure 46 This is a diagram illustrating the conditions for determining the connection topology.

[0059] [ Figure 47 ] Figure 47This shows the method used to derive the device viewpoint distance D. n A diagram of the process.

[0060] [ Figure 48 ] Figure 48 This is a flowchart illustrating the process of determining the connection topology.

[0061] [ Figure 49 ] Figure 49 It is a diagram showing the data structure of the connection topology information.

[0062] [ Figure 50 ] Figure 50 This is a diagram illustrating the data structure of the integrated viewpoint.

[0063] [ Figure 51 ] Figure 51 It is a diagram showing the data structure of public information.

[0064] [ Figure 52 ] Figure 52 This is an illustration of two users viewing a monitor while facing each other.

[0065] [ Figure 53 ] Figure 53 This is a diagram showing the initial state of the LEP mode.

[0066] [ Figure 54 ] Figure 54 This is a diagram illustrating the data structure of the public information according to the second modification.

[0067] [ Figure 55 ] Figure 55 This is a diagram showing the entry structure of the face detection list.

[0068] [ Figure 56 ] Figure 56 This is an illustrative diagram (part 1) illustrating an example of sensing relative position using a combination of a light emitter and an RGB camera.

[0069] [ Figure 57 ] Figure 57 This is an illustrative diagram (part 2) illustrating an example of sensing relative position using a combination of a light emitter and an RGB camera.

[0070] [ Figure 58 ] Figure 58 This is an illustrative diagram showing the collaboration between a device that includes a facial sensing unit and a device that does not have a facial sensing unit.

[0071] [ Figure 59 ] Figure 59This is a hardware configuration diagram illustrating an example of a computer that implements the functionality of a spatial display device. Detailed Implementation

[0072] In the following description, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the following embodiments, the same reference numerals are assigned to the same parts, and redundant descriptions will be omitted accordingly.

[0073] Furthermore, in the following description, when multiple identical parts need to be distinguished from each other, the parts may be numbered in the form of "-m" (where m is a natural number) after the reference numeral indicating the part. This numbering is not performed when the parts do not need to be specifically distinguished.

[0074] In the following text, it is assumed that the information processing device according to the embodiment of this disclosure (hereinafter appropriately referred to as "this embodiment") is a spatial display device 100 (see reference 100). Figure 1 Furthermore, it is assumed that the information processing system according to this embodiment is a display system 1 including multiple spatial display devices 100 (see...). Figure 1 Furthermore, it is assumed that the information processing method according to this embodiment is an information processing method executed by the spatial display device 100.

[0075] The contents of this disclosure will be described in the following order.

[0076] 1. Overview

[0077] 1-1. Display System Configuration Example

[0078] 1-2. Spatial display devices

[0079] 1-3. Usage Forms

[0080] 1-4. Overview of Information Processing Methods

[0081] Implementation methods according to this disclosure

[0082] 2. Configuration Example of Spatial Display Device

[0083] 3. Display system state transitions.

[0084] 4. Processing Procedure

[0085] 4-1. Basic Structure of a Data Packet

[0086] 4-2. Processing in the connected state

[0087] 4-3. Processing steps when the topology is already established

[0088] 4-3-1. Main Equipment

[0089] 4-3-2. Sub-equipment

[0090] 4-4. Data Structures

[0091] 4-5. Absolute position information

[0092] 4-6. Connection Topology Information

[0093] 4-7. Integrating viewpoint information

[0094] 4-8. Public Information

[0095] 5. Modification

[0096] 5-1. First Revision

[0097] 5-2. Second Revision

[0098] 5-3. Third revision

[0099] 5-4. Fourth Revision

[0100] 5-5. Other modifications

[0101] 6. Hardware Configuration

[0102] 7. Conclusion

[0103] 1. Overview

[0104] 1-1. Display System Configuration Example

[0105] Figure 1 This is a diagram illustrating a configuration example of a display system 1 according to an embodiment of the present disclosure. (See diagram for example.) Figure 1 As shown, the display system 1 includes multiple spatial display devices 100-1, 100-2, ..., 100-n (n is a natural number of 2 or greater) and a wireless router 200.

[0106] Spatial display device 100 is a panel-integrated three-dimensional (3D) video display device. Wireless router 200 is a relay of a wireless communication network N. Spatial display devices 100 are configured to communicate with each other via wireless router 200.

[0107] 1-2. Spatial display devices

[0108] Figure 2 This is a diagram illustrating a configuration example of a spatial display device 100 according to an embodiment of the present disclosure. The spatial display device 100 includes a control unit 101, a sensing unit 102, and a 3D panel display unit 103. The spatial display device 100 may also include a speaker (not shown) for audio output.

[0109] The control unit 101 is installed inside the spatial display device 100 and generates a viewpoint video displayed on the 3D panel display unit 103 based on information acquired by the sensing unit 102 and information from the communication unit A5 (described below) included in the spatial display device 100. In this disclosure, the viewpoint video can be considered as a video with motion parallax based on the user's viewpoint position.

[0110] The sensing unit 102 is a component for detecting the user's facial position and relative position with respect to other spatial display devices 100. For example, a wide-angle camera is used for the former, and an ultra-wideband (UWB) sensor is used for the latter. The information detected by the sensing unit 102 is transmitted to the control unit 101, and necessary processing is performed appropriately.

[0111] The 3D panel display unit 103 is a panel-type light-emitting device that includes mechanisms for separating and displaying light beams for the user's left and right eyes. The 3D panel display unit 103 is configured to display stereoscopic images / videos with binocular parallax generated by the control unit 101 through the mechanisms for separating and displaying light beams. That is, in this embodiment, the viewpoint video has motion parallax and binocular parallax depending on the user's viewpoint or facial position. Note that the panel-type light-emitting device according to this disclosure is not limited to a configuration including mechanisms for separating and displaying light beams. That is, the viewpoint video can be a non-stereoscopic image / video without binocular parallax.

[0112] Note that the spatial display device 100 is compact and highly portable. The spatial display device 100 is implemented as, for example, a smartphone or tablet device.

[0113] 1-3. Usage Forms

[0114] Figure 3 This diagram illustrates the usage of display system 1. Display system 1 enables collaborative display of a wide-area scene through multiple spatial display devices 100. As spatial display devices 100 become more compact and highly portable, the opportunity to deploy multiple units in various locations is expected to increase.

[0115] For example, such as Figure 3 As shown, in display system 1, multiple spatial display devices 100 are arranged side-by-side across a wide area in front of the user U's eyes and cooperate to display a wide-area scene. Each spatial display device 100 generates and displays a viewpoint video corresponding to the viewpoint position of the user U obtained therefrom.

[0116] Figure 4 This is a diagram illustrating an example of the intended display of display system 1. Display system 1 collaboratively displays a performance scene, for example, a group of performers as a wide-area scene. Figure 4As shown, display system 1 displays the corresponding members of the performers group in the corresponding spatial display device 100.

[0117] Note that the spatial display device 100 can be positioned and display video with six degrees of freedom (6DoF) relative to the installation space. Therefore, in Figure 4 In the spatial display devices 100-1 to 100-5, the performers do not appear to be in the viewpoint direction of the user U, but rather as if they are actually standing on a stage and performing.

[0118] This display can be achieved by each spatial display device 100 acquiring the facial position of the user U through the sensing unit 102 and generating a video viewed from that direction. However, in the method of each spatial display device 100 individually sensing the facial position of the user U, due to the relative arrangement with respect to the user U, the occurrence of accidental occlusion, etc., there is a high possibility of display failure in some spatial display devices of the spatial display device 100.

[0119] Figure 5 This is an illustration of a typical human field of vision. (For example...) Figure 5 As shown, the field of view θ1 is typically a maximum of about 200°. On the other hand, a range of 70° in front of the person corresponds to a stereoscopic field of view θ2, within which distance estimation using binocular vision is possible.

[0120] In other words, humans can perceive a stereoscopic display within a 70° range in front of them. However, if the perception range of the left or right eye is also included, changes within the field of view can be perceived up to approximately 200°. That is to say, even when spatial display devices are arranged across a 100-span wide area, changes in each display can be perceived, and this is considered to affect the comfort or discomfort of the viewing experience.

[0121] The viewing angle of the camera used by each spatial display device 100 for facial recognition is small relative to the user U's field of view. Therefore, depending on the relative positional relationship between the spatial display device 100 and the user U, the user U's face may deviate from the viewing angle of the camera. It is also foreseeable that the line of sight between the camera and the user U may become obstructed due to various circumstances during viewing.

[0122] This situation is in Figure 6 As shown in the image. Figure 6 This diagram illustrates a situation where viewpoint acquisition is difficult for some spatial display devices in the spatial display device 100. Furthermore, Figure 7 This is a diagram showing an example of an incorrect display of a portion of the text.

[0123] exist Figure 6In the example case, spatial display device 100-1 cannot capture user U within the field of view of the camera device. Furthermore, spatial display device 100-3 cannot correctly detect user U's face because an obstructing object (e.g., a hand) passes in front of the camera device.

[0124] If the information processing method according to this embodiment is not applied, the spatial display devices 100-1 and 100-3 cannot accurately determine the facial position of the user U. Therefore, as Figure 7 As shown, spatial display devices 100-1 and 100-3 produce unnatural displays that are not spatially positioned.

[0125] 1-4. Overview of the information processing method according to the embodiments of this disclosure

[0126] Figure 8 This is a schematic illustration of an information processing method according to an embodiment of the present disclosure. In the information processing method according to this embodiment, such as... Figure 8 As shown, any one of the spatial display devices 100 is set as the "master display device" as the master device, while the other spatial display devices are set as the "sub-display devices" as slave devices.

[0127] exist Figure 8 In the example, spatial display device 100-2 is the main display device "M", while other spatial display devices 100-1 and 100-3 to 100-5 are sub-display devices "S". In the information processing method according to this embodiment, spatial display device 100-1 integrates the position information and viewpoint information of all peripheral sub-display devices "S" to form common information, and distributes the common information to each sub-display device "S".

[0128] Here, spatial display devices 100-4 and 100-5 are sub-display devices "S" capable of acquiring viewpoints, providing their viewpoint information to the main display device "M", and receiving public information integrated by the main display device "M".

[0129] On the other hand, spatial display devices 100-1 and 100-3 are sub-display devices "S" that have difficulty acquiring viewpoints. They provide the main display device "M" with the state related to viewpoint acquisition (acquisition is impossible) and receive public information from the main display device "M".

[0130] The roles of the main display device "M" and the sub-display device "S" change in real time among the spatial display devices 100 due to changes in the surrounding environment, user U's operations, etc. Utilizing this mechanism, even when user U's facial sensing is disabled in some of the spatial display devices 100, viewpoint video correctly positioned relative to space can be displayed in all spatial display devices 100, such as... Figure 4 As shown in the image.

[0131] Therefore, according to the information processing method of this embodiment, collaborative control, including at least collaborative display, can be easily performed by multiple spatial display devices 100. Hereinafter, a configuration example of a display system 1 applying the information processing method of this embodiment will be specifically described.

[0132] 2. Configuration Example of Spatial Display Device

[0133] Figure 9 This is a block diagram illustrating a configuration example of a spatial display device 100 according to an embodiment of the present disclosure. Figure 9 As shown, the spatial display device 100 includes a control unit 101, a sensing unit 102, and a 3D panel display unit 103. In the following text, the spatial display device 100, which is the subject of the description, may be appropriately referred to as the "target device," while other spatial display devices 100 may be referred to as "other devices."

[0134] The control unit 101 includes a face sensing unit A1, a relative position sensing unit A2, an information processing unit A3, and a video processing unit A4. The face sensing unit A1 is a block that acquires an image sequence captured by a camera device attached to a sensing unit 102 in the same direction as the 3D panel display unit 103 and transmits the image sequence to the information processing unit A3 to acquire the face position of the user U.

[0135] The relative position sensing unit A2 is a block that includes sensors for acquiring the relative positional relationship between the spatial display device 100 and a device for transmitting and receiving sensing results. This embodiment assumes the use of a UWB sensor, and therefore, in some cases, a UWB anchoring device may also be used.

[0136] The information processing unit A3 calculates blocks of viewpoint information used by the video processing unit A4 and information transmitted by the communication unit A5 to other spatial display devices 100 (hereinafter appropriately referred to as "other devices") based on information transmitted from the face sensing unit A1 and the relative position sensing unit A2 and information acquired by the communication unit A5. The behavior of the information processing unit A3 varies depending on the master / slave state of the spatial display device 100.

[0137] The video processing unit A4 determines the position of the camera device based on the viewpoint information received from the information processing unit A3, performs rendering processing on the specified content and surrounding scene data, and generates a viewpoint video for panel display. The 3D panel display unit 103 receives and displays the viewpoint video generated by the video processing unit A4.

[0138] Communication unit A5 packages the common information generated by information processing unit A3 and transmits the packaged information. Communication unit A5 is, for example, a transmitter. Furthermore, communication unit A5 receives data packets transmitted from network N to spatial display device 100 (hereinafter appropriately referred to as the "target device"), extracts necessary information, and transmits the information to information processing unit A3.

[0139] Next, the data stream of the spatial display device 100 will be described. Figure 10 This is a diagram illustrating the data flow when the target device is the main display device. Furthermore, Figure 11 This is a diagram showing the data flow when the target device is a sub-display device.

[0140] Note that in Figure 10 and Figure 11 In the diagram, each function block is indicated by a white block, and the information exchanged between function blocks is indicated by a shaded block.

[0141] When the target device is the main display device, such as Figure 10 As shown, face sensing unit A1 serves as face sensing unit C1 and processes face sensing information D1 of the target device. Relative position sensing unit A2 serves as relative position sensing unit C2 and processes the relative position D2 of the target device.

[0142] Information processing unit A3 serves as a connection topology control unit C3, an integrated viewpoint information generation unit C4, a device position coordinate transformation unit C5, a common information generation unit C6, and a slave device information analysis unit C8. Information processing unit A3 processes connection topology information D3, integrated viewpoint information D4, other device facial sensing information D5, other device relative positions D6, and device absolute position information D7.

[0143] The video processing unit A4 serves as the video generation unit C7 and the video display unit C9, and processes the panel video D8. The communication unit A5 serves as the slave device information receiving unit C10 and the public information transmission unit C11, and processes slave device information D9 and public information D10.

[0144] On the other hand, when the target device is a sub-display device, such as Figure 11 As shown, face sensing unit A1 serves as face sensing unit C1 and processes face sensing information D1 of the target device. Relative position sensing unit A2 serves as relative position sensing unit C2 and processes the relative position D2 of the target device.

[0145] Information processing unit A3 serves as a public information analysis unit C12 and a slave device information generation unit C13, and processes connection topology information D3 and integrated viewpoint information D4.

[0146] The video processing unit A4 serves as the video generation unit C7 and the video display unit C9, and processes the panel video D8. The communication unit A5 serves as the slave device information transmission unit C14 and the public information receiving unit C15, and processes slave device information D9 and public information D10.

[0147] When the target device is the main display device, such as Figure 10 As shown, the face sensing unit C1 generates face sensing information D1 of the target device, and the topology control unit C3 and the integrated viewpoint information generation unit C4 receive the face sensing information D1 of the target device.

[0148] The relative position sensing unit C2 generates the relative position D2 of the target device, and the device position coordinate transformation unit C5 receives the relative position D2 of the target device. The connection topology control unit C3 generates connection topology information D3 based on the face sensing information D5 of other devices and the absolute position information D7 of the devices, and the common information generation unit C6 receives the connection topology information D3. The connection topology information D3 is information related to the topology management of the main display device and the sub-display devices in the network N of the display system 1.

[0149] The integrated viewpoint information generation unit C4 generates integrated viewpoint information D4, which is received by the common information generation unit C6 and the video generation unit C7. The device position coordinate transformation unit C5 generates absolute device position information D7 based on the relative position D2 of the target device and the relative positions D6 of other devices, and receives the absolute device position information D7 by the topology control unit C3 and the common information generation unit C6.

[0150] The public information generation unit C6 generates public information D10 based on the connection topology information D3, the integrated viewpoint information D4, and the device absolute position information D7, and the public information transmission unit C11 receives the public information D10. The video generation unit C7 generates panel video D8 based on the target device facial sensing information D1 and the integrated viewpoint information D4, and the video display unit C9 receives the panel video D8 and performs viewpoint video processing on the 3D panel display unit 103.

[0151] The device information analysis unit C8 generates the relative positions D6 of other devices based on the device information D9, and the device position coordinate transformation unit C5 receives the relative positions D6 of other devices. The device information receiving unit C10 receives the device information D9 from the sub-display device through network N and transmits it to the device information analysis unit C8. The public information transmission unit C11 transmits the public information D10 to the sub-display device through network N.

[0152] When the target device is a sub-display device, such as Figure 11As shown, the face sensing unit C1 generates the target device face sensing information D1, and the video generation unit C7 and the device information generation unit C13 receive the target device face sensing information D1.

[0153] The relative position sensing unit C2 generates the relative position D2 of the target device and receives the relative position D2 of the target device from the device information generation unit C13. The video generation unit C7 generates panel video D8 based on the facial sensing information D1 of the target device and the integrated viewpoint information D4, and the video display unit C9 receives the panel video D8 and performs viewpoint video processing on the 3D panel display unit 103.

[0154] The public information analysis unit C12 analyzes the public information based on public information D10 and outputs the analysis results to the connection topology information D3 and the integrated viewpoint information D4. The slave device information generation unit C13 generates slave device information D9 based on the target device facial sensing information D1 and the target device relative position D2, and transmits it to the slave device information transmission unit C14.

[0155] The device information transmission unit C14 transmits the slave device information D9 to the main display device via network N. The public information receiving unit C15 receives public information D10 from the main display device via network N and transmits it to the public information analysis unit C12.

[0156] 3. Display system state transitions.

[0157] Next, the state transitions of system 1 will be described. Figure 12 This is a state transition diagram of system 1.

[0158] like Figure 12 As shown, in display system 1, spatial display devices 100 have two states: "connected" and "topology established." The "connected" state refers to a state where the master / slave status of each spatial display device 100 is not determined in display system 1. The "topology established" state refers to a state where at least one master device is determined in display system 1. n It is a transition between states, and it is based on... Figure 12 The transformation of the occurrence conditions shown.

[0159] 4. Processing Procedure

[0160] 4-1. Basic Structure of a Data Packet

[0161] Next, we will refer to Figures 13 to 25 Describe in sequence Figure 12 The processing procedure for each state shown is as follows. First, the basic structure of the data packets exchanged between the spatial display devices 100 in the display system 1 will be described. Figure 13This is an illustration of the basic structure of a data packet (part 1). Figure 14 This is an illustration of the basic structure of a data packet (part 2).

[0162] Note that in the following text, the main display device may be referred to as the "main device". Additionally, the sub-display device may be referred to as the "sub-device".

[0163] like Figure 13 As shown, the data packets exchanged in display system 1 include requests (REQ) with four “msgType” types and responses (RES) with three “rspType” types.

[0164] The msgType "001" with "MAIN_DISCOVERY" corresponds to REQ1. REQ1 is a discovery request used to discover the master device. Note that REQ1 corresponds to the example of "a first request asking whether the master display device exists or not".

[0165] The msgType "002" "Sub-registration (SUB_REGISTRATION)" corresponds to REQ2. REQ2 is a sub-device registration request, which is a request used to register as a sub-device. Note that REQ2 corresponds to the example of "Second Request".

[0166] The msgType "003" "Status Report (STATUS_REPORT)" corresponds to REQ3. REQ3 is a report about the status of the sub-device, which is transmitted from the sub-device to the master device.

[0167] The "Status Push (STATUS_PUSH)" in msgType "004" corresponds to REQ4. REQ4 is information transmitted from the master device to the slave device.

[0168] Figure 14 The data structures for each pair of REQ1 to REQ4 and their corresponding RES1 to RES4 are shown in order from top to bottom. Figure 14 As shown, REQ3 includes device information D9 as a report on the status of the sub-device. Additionally, REQ4 includes public information D10 that is generated periodically or as needed in the master device. The processing procedures for exchanging these data packets will be described below.

[0169] 4-2. Processing in the connected state

[0170] First, the processing procedure when the device is in a connected state will be described. Figure 15 This is a flowchart illustrating the processing procedure when the device is in a connected state. Figure 16 Is with Figure 15The corresponding processing sequence diagram.

[0171] When connected, the control unit 101 of the spatial display device 100 first acquires the target device's facial sensing information D1 and the target device's relative position D2 (step S101).

[0172] Then, the control unit 101 broadcasts REQ1 as a discovery request (step S102). Subsequently, the control unit 101 determines whether it has received a response RES1 from the master device (step S103).

[0173] Upon receiving the response RES1 (Yes in step S103), the control unit 101 generates slave device information D9 based on the target device facial sensing information D1 and the target device relative position D2 (step S104). Then, the control unit 101 transmits the slave device registration request REQ2 to the master device (step S105).

[0174] Then, after waiting for acceptance from the master device (step S106), a determination is made as to whether response RES2 has been received (step S107). If response RES2 is not received (No in step S107), the control unit 101 repeats the process from step S106. If response RES2 is received (Yes in step S107), the control unit 101 changes the status to Topology Established (step S108) and ends the process.

[0175] On the other hand, if no response RES1 is received in step S103 (No in step S103), the control unit 101 designates the target device as the master device and causes the target device to perform tasks as the master display device (step S109). Tasks as the master display device include, for example, generating connection topology information D3, transmitting a response indicating that the master device has been set, registering the sub-device in the connection topology information D3, updating the connection topology information D3, and generating and updating common information D10. Then, the control unit 101 converts the relative position D2 of the target device into absolute position information D7 using the device position coordinate conversion unit C5 (step S110).

[0176] Then, the control unit 101 transmits the target device facial sensing information D1 and the device absolute position information D7 to the connection topology control unit C3, and generates connection topology information D3 (step S111). Then, the control unit 101 changes the status to "topology established" (step S108) and ends the process. Note that, with... Figure 15 The processing sequence diagram corresponding to the processing procedure is as follows: Figure 16 As shown in the image.

[0177] 4-3. Processing steps when the topology is already established

[0178] 4-3-1. Main Equipment

[0179] Next, we will describe the process of acting as the master device when the topology has been established. Figures 17 to 21 These are flowcharts (part 1) to (part 5) showing the processing procedures of the master device in the state where the topology has been established. Figure 22 Is with Figures 17 to 21 The corresponding processing sequence diagram.

[0180] To summarize the process, the basic operations consist of the following: receiving requests from sub-devices in the message processing loop, performing necessary processing, and sending the results back to the requesting sub-device. Within the message processing loop, a common message D10 is generated, and the status is pushed to all sub-devices at regular time intervals.

[0181] In the generation of public information D10, the connection topology information D3 is recalculated. However, due to changes in various conditions in the installation environment, the main device ID (mainDeviceId) may be selected differently from the target device. In this case, the target device is changed to a sub-device, the process for the main device ends, and the process for the sub-device, where the state topology has been established, restarts. Waiting to receive request REQ4 is an exception handling condition, and if the request is received, the main device (DUPLICATED_MAIN) is repeated as an error return.

[0182] Specifically, such as Figure 17 As shown, when the state transitions from the connected state to the topology established state, the control unit 101 starts the message processing loop of the data packet (step S201).

[0183] In this message processing loop, control unit 101 determines whether request REQ1 has been received (step S202). If request REQ1 has been received (yes in step S202), control unit 101 performs main discovery processing (step S203), and then proceeds to step S210.

[0184] If no request REQ1 is received (No in step S202), the control unit 101 determines whether request REQ2 has been received (step S204). If request REQ2 is received (Yes in step S204), the control unit 101 performs a sub-registration process (step S205), and then proceeds to step S210.

[0185] If no request REQ2 is received (No in step S204), the control unit 101 determines whether request REQ3 has been received (step S206). If request REQ3 is received (Yes in step S206), the control unit 101 performs status report processing (step S207), and then proceeds to step S210.

[0186] If request REQ3 is not received (No in step S206), control unit 101 determines whether request REQ4 has been received (step S208). If request REQ4 is received (Yes in step S208), control unit 101 transmits a duplicate message to the source of request REQ4 (step S209), and then proceeds to step S210.

[0187] If no request REQ4 is received (No in step S208), the control unit 101 proceeds directly to step S210. In step S210, the control unit 101 executes status push processing #1 (step S210).

[0188] Subsequently, the control unit 101 performs viewpoint video processing in the video processing unit A4 based on the target device's facial sensing information D1 and the integrated viewpoint information D4 (step S211). Then, it determines whether the message processing loop has ended (step S212). If the message processing loop has not ended (No in step S212), the processing is repeated from step S202. If the message processing loop has ended (Yes in step S212), the processing ends.

[0189] Next, the main discovery process will be described. For example... Figure 18 As shown, in the main discovery process, the control unit 101 sets the display ID (displayId) of the target device in the main display ID (mainDisplayId) of the data packet (step S301).

[0190] Then, the control unit 101 sends a response RES1 to the source of the request REQ1 (step S302) and ends the main discovery process.

[0191] The main discovery process will now be described. For example... Figure 18 As shown, in the main discovery process, the control unit 101 sets the display ID of the target device in the main display ID of the data packet (step S301).

[0192] Then, the control unit 101 sends a response RES1 to the source of the request REQ1 (step S302) and returns from the main discovery process.

[0193] Next, the sub-registration process will be described. For example... Figure 19As shown, in the sub-registration process, the control unit 101 adds the display ID of request REQ2 to the display ID list (step S401).

[0194] Then, the control unit 101 sends a response RES2 to the source of the request REQ2 (step S402) and returns from the sub-registration process.

[0195] Next, the status report processing will be described. For example... Figure 20 As shown, in the status report processing, the control unit 101 updates the entry in the device status list (deviceStatusList) corresponding to the display ID of the request REQ3 with the slave device information D9 of the request REQ3 (step S501).

[0196] Then, the control unit 101 sends a response RES3 to the source that requested REQ3 (step S502) and returns from the status report processing.

[0197] Next, we will describe the status push processing #1. For example... Figure 21 As shown, in status push processing #1, the control unit 101 inputs the device status list to the information processing unit A3 and outputs the connection topology information D3, the integrated viewpoint information D4 and the device absolute position information D7 (step S601).

[0198] Then, the control unit 101 generates common information D10 based on the connection topology information D3, the integrated viewpoint information D4 and the device absolute position information D7 (step S602), and stores it in the response RES4 (step S603).

[0199] Then, the control unit 101 broadcasts a request REQ4 to the destination in the displayed ID list (step S604). Subsequently, the control unit 101 determines whether the master device ID of the connection topology information D3 is not the device ID of the target device (step S605).

[0200] If the device ID is not the device ID of the target device (Yes in step S605), the control unit 101 sets the target device as a sub-device, switches to the topology established state (step S606), and ends the process.

[0201] If the device ID is the device ID of the target device (No in step S605), the control unit 101 determines whether the resType of the response RES4 is a repeating master (step S607).

[0202] If resType is "Duplicate Master" (Yes in step S607), control unit 101 switches to the connected state and ends the process. If resType is not "Duplicate Master" (No in step S607), control unit 101 returns from status push process #1. Note that, according to Figures 17 to 21 The processing sequence diagram of the processing procedure is as follows: Figure 22 As shown in the image.

[0203] 4-3-2. Sub-equipment

[0204] Next, we will describe the process of handling a sub-device when the topology has been established. Figure 23 and Figure 24 These are flowcharts (part 1) and (part 2) illustrating the processing procedure for a sub-device when the topology has been established. Figure 25 Is with Figure 23 and Figure 24 The corresponding processing sequence diagram.

[0205] To outline this process, if the master device ID in the connection topology information D3 included in the public information D10 matches the device ID of the target device, the target device is designated as the new master device through the determination of the master device. The matching of the master device ID in the connection topology information D3 with the device ID of the target device means that the public information D10 includes identification information indicating the master display device (the device ID of the target device). In this case, the target device's role changes to master device, its task as a sub-device ends, and its task as a master device restarts once the topology is established.

[0206] Note that if a certain period of time has passed without receiving a status push from the master device, the master device is considered to be in a state where it cannot continue processing for some reason, and the state drops to connected.

[0207] Specifically, such as Figure 23 As shown, when the state transitions from the connected state to the topology established state, the control unit 101 starts the message processing loop of the data packet (step S701).

[0208] In this message processing loop, the control unit 101 inputs the target device facial sensing information D1 and the target device relative position D2 to the slave device information generation unit C13 to generate slave device information D9 (step S702).

[0209] Then, the control unit 101 generates request REQ3 based on the slave device information D9 and transmits it to the master display ID (that is, the master device) (step S703). Then, the control unit 101 determines whether request REQ4 has been received (step S704).

[0210] If request REQ4 is received (Yes in step S704), status push processing #2 is executed (step S705), and then the process transitions to step S708. If request REQ4 is not received (No in step S704), control unit 101 increments the counter (step S706) and determines whether the counter exceeds the threshold (step S707).

[0211] If the counter is equal to or less than the threshold (No in step S707), the control unit 101 performs viewpoint video processing in the video processing unit A4 based on the viewpoint information of the target device face sensing information D1 and the integrated viewpoint information D4 (step S708), determines whether the message processing loop has ended (step S709), if the message processing loop has not ended (No in step S709), the processing is repeated from step S702, and if the message processing loop has ended (Yes in step S709), the processing ends.

[0212] On the other hand, if the counter exceeds the threshold (Yes in step S707), the control unit 101 switches to the connected state (step S710) and ends the process.

[0213] Next, we will describe status push processing #2. For example... Figure 24 As shown, in the status push processing #2, the control unit 101 processes the public information D10 of request REQ4 through the public information analysis unit C12, and extracts the connection topology information D3 and the integrated viewpoint information D4 (step S801).

[0214] Then, the control unit 101 determines whether the master device ID of the connection topology information D3 is the device ID of the target device (step S802).

[0215] If the device ID is the same as the target device's device ID (Yes in step S802), the control unit 101 sets itself as the master device, switches to the topology established state (step S803), and ends the process.

[0216] If the device ID is not the device ID of its own device (No in step S802), the control unit 101 resets the counter (step S804) and returns from the status push process #2. Note that, with Figure 23 and Figure 24 The processing sequence diagram corresponding to the processing procedure is as follows: Figure 25 As shown in the image.

[0217] 4-4. Data Structures

[0218] Next, Figures 26 to 29 The data structure in the display system 1 according to this embodiment is shown. Figures 26 to 29 These are diagrams (part 1) to (part 4) illustrating the data structure of the display system 1 according to an embodiment of the present disclosure.

[0219] Figure 26 The data structure for face detection, which is obtained by each sub-device or integrated by the master device, is shown.

[0220] also, Figure 27 The data structure of the local geometry of the relative positions of other devices acquired by each spatial display device 100 is shown. Note that in the case where two or more other devices are positioned within a detectable range around a device, multiple acquisition results will be obtained for this data.

[0221] Figure 28 The data structure of slave device information D9 (subDeviceInfo) transmitted from each of the sub-devices to the master device is shown. At the end, the local geometry list (localGeometryList) contains entries representing the relative local geometry of other devices.

[0222] Figure 29 The data structure for the deviceStatus of each device is shown, which is an entry in a list of device statuses managed by the master device.

[0223] 4-5. Absolute position information

[0224] Next, the process until the absolute position information D7 of the device is obtained from the local geometric information acquired by each spatial display device 100 will be described. First, the mechanism by which each sub-device acquires local geometric information will be described.

[0225] Figure 30 This is a diagram showing an example of equipment arrangement. Note that below, the equipment ID of equipment 100 is displayed according to each space. The equipment with equipment ID 01 can be referred to as "equipment 01".

[0226] Figure 30 The example shown is that sub-device 01 and sub-device 03 are respectively located on the left front side and right front side of sub-device 05.

[0227] Each sub-device uses the UWB protocol as described above as a way to detect the relative position of other devices. Figure 31 This is an illustration of relative position detection using UWB. Figure 31 This demonstrates how to detect distance and direction from other devices using the corresponding UWB protocol.

[0228] Each device obtains its distance to other devices using either the UWB Time of Arrival (ToA) protocol or the Time Difference of Arrival (TDoA) protocol. Next, using the Angle of Arrival (AoA) protocol, the direction of arrival of the detected wave is obtained as the relative angle to the normal to the sensor surface.

[0229] Using the obtained distance and direction, the relative position vector with respect to the target device is uniquely determined. Furthermore, these detection results are obtained bidirectionally using paired symmetrical devices, and thus the relative position information M is obtained by integrating the detection results from both methods. RT_P_Q It is the relative position matrix from device P to device Q.

[0230] Note that UWB's location detection range is limited to a radius of approximately 0.5 m around the device. Therefore, when a large number of devices are deployed, the detection results are localized. Figure 32 This is a diagram showing an example of the UWB detection range.

[0231] Figure 32 This illustrates the scenario where the detection range extends to a non-overlapping extent. Figure 32 In this setup, 12 devices are positioned in an area approximately 2.0 m horizontally and 1.0 m vertically. As described above, the detection range of UWB is a circle with a radius of approximately 0.5 m around each device, ensuring that a device positioned in the right or left half of this area cannot directly detect a device positioned in another area.

[0232] In the following text, we will assume that only five devices are deployed, that is, Figure 32 The subsequent processing procedure is described in the case of devices 01, 02, 03, 06, and 07 in the upper left part. Figures 33 to 37 The hypothetical detection results for each device are shown in the figure.

[0233] Figure 33 This is a diagram showing the detection results of the local geometric information of device 01. Figure 34 This is a diagram showing the detection results of the local geometric information of device 02. Figure 35 This is a diagram showing the detection results of the local geometric information of device 03. Figure 36 This is a diagram showing the detection results of the local geometric information of device 06. Figure 37 This is a diagram showing the detection results of the local geometric information of device 07.

[0234] also, Figure 38 It is shown Figures 33 to 37 The detection results are plotted against a graph showing the relative position information. As mentioned above, detection is performed bidirectionally, and therefore, as... Figure 38 As shown, in Figures 33 to 37All detection results contain pairs of detection results in the direction from device P to device Q and detection results in the direction from device Q to device P.

[0235] Associated with these test results, in Figure 38 In the example, relative position information M is obtained. RT_1_2 M RT_1_6 M RT_2_3 M RT_2_6 M RT_2_7 M RT_3_7 and M RT_6_7 .

[0236] Relative position information M RT_P_Q This information indicates the relative translation and rotation of device Q when viewed from device P, and is represented by a 4×4 matrix or a combination of three-dimensional vectors and quaternions. The methods used to derive a specific relative position (translation and rotation) are described below.

[0237] Figure 39 This is a graph illustrating the relationship between the relative positions of the devices and the detection results of local geometric information. The coordinate systems xy and x′y′ are the device-specific coordinate systems of devices P and Q, respectively, with O and O′ being their respective origins. The local geometric information detected by the position sensors of each device—that is, the positions and orientations of other devices—is represented by values ​​in this coordinate system.

[0238] Position detection is performed mutually, and therefore Figure 38 The position vector V of the detection device Q in the middle device P P_Q And the position vector V of device Q and device P. Q_P These are identical vectors with different directions in an absolute coordinate system. On the other hand, the rotation angle R of the corresponding vector relative to the specific coordinate system of the corresponding device... _P and R _Q They have different values.

[0239] Then, obtain V in the coordinate system xyz. P_Q Rotation matrix M in the direction R_P The rotation matrix can be generated from the unit vectors of the orthogonal coordinate axes, and when the y-axis is defined at the same angle as V... P_Q When the z-axis is defined in the upward direction on the paper surface, it is obtained by the following expressions (a) to (d).

[0240] [Mathematical Expression 1]

[0241]

[0242] [Mathematical Expression 2]

[0243]

[0244] [Mathematical Expression 3]

[0245]

[0246] [Mathematical Expression 4]

[0247]

[0248] Note that this article assumes all devices are arranged on the same plane, such as a table, and therefore the z-axis of the device coordinate system may be common. With all three axes (x, y, z) shifted for each device, the z-axis calculation also needs to be performed by, for example, referring to the values ​​of the inertial measurement unit (IMU) equipped in each device.

[0249] Similarly, -V in coordinate system x′y′z′ is obtained through the following expressions (e) to (h). Q_P Rotation matrix M in the direction R_Q .

[0250] [Mathematical Expression 5]

[0251]

[0252] [Mathematical Expression 6]

[0253]

[0254] [Mathematical Expression 7]

[0255]

[0256] [Mathematical Expression 8]

[0257]

[0258] Here, V P_Q Not used as originally intended. The reason is V. P_Q In the absolute coordinate system, with -V Q_P They coincide, but have different values ​​in the device-specific coordinate system.

[0259] Then, by means of the following expression (i) using M R_Q and M R_Q To calculate the rotation M of the x′y′z′ coordinate system relative to the xyz coordinate system. R_P_Q .

[0260] [Mathematical Expression 9]

[0261]

[0262] Furthermore, when the translation component VP_Q The x, y, and z components are defined as t x t y and t z And M R_P_Q The elements of row m and column n are defined as a mn At that time, the final 4×4 rotation / translation matrix M RT_P_Q It can be written as the following expression (j).

[0263] [Mathematical Expression 10]

[0264]

[0265] When performing these calculations on all pairs, one can utilize the corresponding method as shown in... Figure 40 A set of relative position matrices in the graph structure. Figure 40 This is a diagram showing the structure of a relative position map. Figure 40 In a graph structure, even with the devices used as the start and end points determined, there may be multiple paths.

[0266] exist Figure 40 There are four paths from device 06 to device 02 (06-01-02, 06-02, 06-07-02, 06-07-03-02). In the generation of device absolute position information D7, errors between these paths are removed by selection or optimization.

[0267] The device absolute position information D7 is the position (translation and rotation) information of each device based on the device-specific coordinate system of the master device. This is achieved by sequentially tracing the path from the master device, which serves as the base point, to each device. Figure 40 The graph in the image, and the relative position information M is integrated in this process. RT_P_Q And obtained.

[0268] Figure 41 It is a graph that illustrates the data structure of the graph structure. Figure 42 This is a diagram illustrating an example of how graph data is stored. Figure 43 This is a diagram showing the data structure of devPosRotEntry, which illustrates the relative position information between devices.

[0269] Figure 40 The relative position diagram structure in the middle is packaged into Figure 41 The data format of the graph structure shown is stored in the connection topology information D3 described below. As an example, Figure 42 It shows the storage Figure 40 The data content of the graph structure in the context of the graph.

[0270] and Figure 40The relative positional relationship of each edge in the diagram is determined by... Figure 43 The device absolute location information D7 shown represents the device-to-device relative location information devPosRotEntry. This data is stored as a list in public information D10, which has... Figure 41 The number of nEdge entries in the graph structure is the same as the number of entries.

[0271] 4-6. Connection Topology Information

[0272] Next, the connection topology information D3 will be described. The connection topology information D3 is information related to the allocation of master and slave devices for each spatial display device 100. Figure 44 This is a graph showing the initial branches determined by the connection topology.

[0273] Figure 44 This illustrates a rough flow of states branching from the initial state. (For example...) Figure 44 As shown, consider the case where the target device broadcasts a discovery request and waits for a response for a specific period of time, then 1) only the target device exists, and 2) the master device already exists.

[0274] If only the target device exists, it may be used alone, or it may be the first device in a process of deploying multiple other devices. In this case, the target device is set as the master device, the state transitions to "topology established," and the process ends.

[0275] On the other hand, if the master device already exists, the latest state of the target device is packaged, and a sub-registration is sent to request registration as a sub-device from the master device. After receiving a reply indicating that registration is complete, the target device is set as a sub-device, the state transitions to topology established, and the process ends.

[0276] Note that even after the master and slave devices are assigned during the initial transition, the master device updates the connection topology information D3 at regular intervals. The details of the update process for the connection topology information D3 will be described below.

[0277] First, assume that five devices, namely device 01, device 02, device 03, device 06 and device 07, are set up and are in different states relative to the environment and user U. Figure 45 This is a diagram illustrating an example of the additional conditions during connection topology determination.

[0278] exist Figure 45 In the example, an obstructing object (hand) exists between device 01 and user U, obstructing user U's facial position sensing. Furthermore, enhanced illumination is applied to devices 02 and 03, and the impact of this situation on user U's facial position sensing is also considered. Taking these additional conditions into account, in... Figure 46 The diagram shows the relevant parameters for each device referenced in the connection topology determination. Figure 46 This is a diagram illustrating the conditions for determining the connection topology.

[0279] The "status" and "trustworthiness" of the sub-devices are obtained from the face detection and sub-device information included in the sub-device information D9, which is transmitted to the master device in the "status report" request REQ3 by each device. The master device obtains equivalent information from the target device's face sensing information D1.

[0280] “D n " is the distance from the origin of the target device's device-specific coordinates to the user U's viewpoint (device viewpoint distance), and through..." Figure 47 The process shown is used to obtain it. Figure 47 This shows the method used to derive the device viewpoint distance D. n The process diagram. Note that when device 06 is the master device, V pos_6 It is the origin (0, 0, 0).

[0281] also, Figure 46 The content of the connection topology determination conditions shown is determined by the connection topology control unit C3 as having Figure 29 The device status shown in the figure is managed as a list of device status entries.

[0282] Specifically, when the target device begins its role as the master device, the connection topology control unit C3 generates a device status list, and when the role ends, the device status list is discarded. Furthermore, each time the target device receives a status report from a sub-device, the corresponding device ID entry is recalculated and updated.

[0283] Using the device status list managed and updated as described above, the connection topology control unit C3 asynchronously generates connection topology information D3. The processing procedure is as follows: Figure 48 As shown in the image. Figure 48 This is a flowchart illustrating the process of determining the connection topology. Figure 49 This is a diagram showing the data structure of the connection topology information D3.

[0284] The connection topology control unit C3 first reads the device status list (step S901). Then, the connection topology control unit C3 excludes devices with a false (NG) status (step S902).

[0285] Then, it is determined whether the number of entries is 0 (step S903). If there is not even a single target device at this time, an error will be displayed, indicating that no valid device exists. Specifically, if the number of entries is 0 in step S903 (yes in step S903), the connection topology control unit C3 displays an error such as "No valid device found" (step S904).

[0286] Then, the connection topology control unit C3 determines whether to end the process (step S905) and confirms that the process continues. If the process is not to end ("No" in step S905), the system waits for a specific period of time (step S906), and then repeats the process from step S901. When it is time to end the process ("Yes" in step S905), the device is terminated.

[0287] Furthermore, if the number of entries in step S903 is not zero (no in step S903), that is, if there are valid entries, the connection topology control unit C3 calculates the sorting key S for each entry. n (Step S907). Note that the connection topology control unit C3 calculates the sorting key S using the following expression (k). n .

[0288] [Mathematical Expression 11]

[0289]

[0290] Note that, as Figure 45 As shown, in the case of device 01, the hand is placed in a position that obstructs the field of view of the camera device. Furthermore, strong illumination is applied to the panel surfaces of devices 02 and 03, which is believed to cause a decrease in the confidence level.

[0291] When used for Figure 46 The above expression (k) for the valid entry (state = OK) in the data is obtained by using "D n When performing calculations using "and" and "credibility", the sorting key S of device 06 n It has the highest value of "1.00". In this case, the topology control unit C3 selects device 06 as the master device.

[0292] Note that the connection topology control unit C3 basically selects the device that can sense user U and is closest to user U as the master device, but reduces the priority of devices that have "short distance but low accuracy in facial recognition" by weighting based on confidence.

[0293] Then, the connection topology control unit C3 substitutes the device ID of the device ultimately selected as the master device into the master device ID, and compares the result with... Figure 41The graph structure is stored together in Figure 49 The connection topology information D3 shown is included.

[0294] In other words, such as Figure 48 As shown, after executing step S907, the topology control unit C3 is connected with the sorting key S. n The entries are sorted in descending order (step S908). Then, the connection topology control unit C3 sets the device ID of the first sorted entry as the master device ID (step S909) and ends the process.

[0295] 4-7. Integrating viewpoint information

[0296] Next, we will describe the integration of viewpoint information D4. Figure 50 This is a diagram illustrating the data structure of the integrated viewpoint information D4. For the integrated viewpoint information D4, the control unit 101 (integrated viewpoint information generation unit C4) first transforms the viewpoint of each acquired device to the master device coordinate system reference by using a graph matrix set.

[0297] When the facial coordinate vector before the transformation of device n is defined as faceVec n Furthermore, the transformed facial coordinate vector is defined as faceVec. n At that time, the calculations in each device 01, 02, 03, 06 and 07 are as shown in the following expressions (l) to (p).

[0298] [Mathematical Expression 12]

[0299]

[0300] [Mathematical Expression 13]

[0301]

[0302] [Mathematical Expression 14]

[0303]

[0304] [Mathematical Expression 15]

[0305]

[0306] [Mathematical Expression 16]

[0307]

[0308] Furthermore, when the face detection confidence (credibility) of each device n is used as the weight w i When the weighted average of the obtained viewpoints is calculated using the following expression (q), the integrated viewpoint faceVec (with an overline) is obtained.

[0309] [Mathematical Expression 17]

[0310]

[0311] Integrating the viewpoint faceVec (with an overline) and each M RT_P_Q As public information D10, the status is pushed to each device, allowing the device-specific coordinates on each device side to be returned via the following expressions (r) to (v).

[0312] [Mathematical Expression 18]

[0313]

[0314] [Mathematical Expression 19]

[0315]

[0316] [Mathematical Expression 20]

[0317]

[0318] [Mathematical Expression 21]

[0319]

[0320] [Mathematical Expression 22]

[0321]

[0322] By using faceVec n Even when device n cannot perform face sensing, it can reproduce panel video that accurately reflects the user U's viewpoint. Furthermore, in obtaining faceVec n During the process of '', all matrices to the left of faceVec (with the overline) are integrated, but the rotation component of the integrated result can be combined with Figure 50 The integrated viewpoint information D4 shown is associated with faceRot (roll / pitch / yaw).

[0323] The control unit 101 thus obtains information about the facePos and faceRot of the master device, and stores the facePos and faceRot in the memory. Figure 50 The integrated viewpoint information D4 is shown.

[0324] 4-8. Public Information

[0325] Next, we will describe public information D10. Figure 51This is a diagram illustrating the data structure of public information D10. For public information D10, control unit 101 (public information generation unit C6) integrates connection topology information D3, viewpoint information D4, and device absolute position information D7, and stores them in... Figure 51 The public information shown is in D10.

[0326] The public information D10 is data transmitted from the master device to the slave device via status push.

[0327] 5. Modification

[0328] Note that although embodiments of the present disclosure have been described, the information processing method according to the present disclosure also includes several modifications in addition to the above.

[0329] 5-1. First Revision

[0330] First, the first modification will be described. In this first modification, instead of acquiring facial images via an RGB camera device, the face sensing unit C1 can combine an optical marker worn around the head by the user U with an infrared camera device that detects the optical marker. In this case, the positions of the head and eyes are estimated as three-dimensional rigid bodies with 6 DoF degrees of freedom.

[0331] 5-2. Second Revision

[0332] Next, a second modification will be described. In this second modification, the viewpoint positions of two or more users U can be shared using the communication method of this embodiment. Figure 52 This is an illustration of two users viewing a monitor while facing each other. Figure 52 The illustration depicts two users, U-1 and U-2, each holding spatial display devices 100-1 and 100-2 respectively, facing each other to view the other user's screen. Spatial display device 100-1 acquires user U-2's Face2 as viewpoint information and displays a viewpoint video with motion parallax based on user U-2's Face2 viewpoint information. Conversely, spatial display device 100-2 acquires user U-1's Face1 as viewpoint information and displays a viewpoint video with motion parallax based on user U-1's Face1 viewpoint information.

[0333] In this scenario, it is desirable for each spatial display device 100 to use a separately detected viewpoint location, rather than a shared viewpoint. In the second modification, the operating mode used to handle this situation is referred to as Local Viewpoint (LEP) mode.

[0334] Notice, Figure 52Examples are shown where spatial display devices 100-1 and 100-2 display characters corresponding to users U-1 and U-2, respectively. In this disclosure, a "character" is a type of virtual object that can operate autonomously and may be referred to as an avatar or proxy. Although... Figure 52 Examples of humanoid characters displayed on spatial display devices 100-1 and 100-2 are disclosed, but the technology disclosed herein is not limited thereto. The character displayed on spatial display device 100 may be an animal or a movable object such as a robot.

[0335] Figure 53 This diagram illustrates the initial state of LEP mode. First, user U-1 places spatial display device 100-1 on a table. With no other devices nearby, spatial display device 100-1 becomes the master device and enters a standby loop involving sub-devices.

[0336] The main device itself cannot use a viewpoint, and therefore there are no valid viewpoints exceeding the threshold in the topology information of connection topology information D3, and the image at the default viewpoint position is displayed on the panel. In this document, user U-2 participates in spatial display device 100-2 and begins information sharing according to the second modification.

[0337] Spatial display device 100-1 is the master device, and spatial display device 100-2 is the slave device, and relative information position detection is performed via UWB. Although status reporting from the slave device to the master device and status pushing from the master device to the slave device are also performed according to this embodiment, the following points differ in the case of LEP.

[0338] First, in LEP mode, viewpoint video for each spatial display device 100 is generated using sensors of the spatial display devices. Therefore, the integrated viewpoint information D4 is not calculated, and instead, face detection included in the status report of each device is listed and included in the public information D10a according to the second modification.

[0339] Figure 54 This is a diagram illustrating the data structure of the public information D10a according to the second modification. The face detection list is a list of pairs of device IDs and face detection information for each device. The entry structure of this list is as follows: Figure 55 As shown in the image. Figure 55 The entry structure of the face detection list is shown.

[0340] Second, in LEP mode, during viewpoint video generation, Figure 17 Step S211 shown or Figure 23The step S708 shown is changed to a process of "generating a viewpoint video based on the face detection information of the target device shared in the face sensing information D1 or public information D10a of the target device".

[0341] Third, in LEP mode, the viewpoint position of user U of another device that cannot be detected by the target device can be determined by referring to viewpoint information of other devices shared via status push. Therefore, for example, it can be possible to display... Figure 52 The character on the spatial display device 100-1 performs actions or dialogues related to the gaze direction or facial position of the user U-1 who is actually behind the character.

[0342] As an example of dialogue based on the facial position of user U-1, spatial display device 100-1 can control a speaker so that the character can utter phrases such as, "Master, please stand directly behind me and do not leave my left rear." As an example of action based on the facial position of user U-1, spatial display device 100-1 can change the orientation of the character's face so that it faces the area behind spatial display device 100-1, and manipulate the character to look at the facial position of user U-1.

[0343] In LEP mode, in addition to the user's viewpoint information, the target device can also obtain the location information of real objects outside the detection range of the target device's sensors from another device. The target device can determine the location information of real objects that cannot be detected by the target device by referring to the location information of real objects obtained by other devices. These real objects are, for example, real objects located behind the target device other than the user, such as a table or chair. For example, this allows the display to... Figure 52 The character on the spatial display device 100-1 performs actions or dialogues related to real objects that are outside the detection range of the sensors on the spatial display device 100-1.

[0344] 5-3. Third revision

[0345] Next, the third modification will be described. Figure 56 and Figure 57 These are illustrative diagrams (part 1) and (part 2) illustrating an example of sensing relative position using a combination of a light emitter and an RGB camera. In the third modification, each spatial display device 100 includes a light emitter and an RGB camera. The light emitter is disposed, for example, on the rear surface of each device.

[0346] In the third modification, the relative position sensing unit C2 obtains the relative position by capturing the light emission of the light emitter using an RGB camera device. Figure 56The image capture of the spatial display device 100 ("2" and "4") shows the light emission of the light emitter of the spatial display device 100 ("1").

[0347] In this scenario, for example, the light emitter at the top of the housing is controlled to emit light via timing, and RGB cameras in all units capture images. The relative positions between the units are then determined based on analysis of the bright image and timing information. Figure 57 As shown in the left figure, at the timing when the light emitter of a specific device (here, "device 01") emits light, device 04 behind device 01 captures an image including the light emitter (see reference). Figure 57 (See the right image in the image). By analyzing this image using methods such as perspective n-point (PnP), the 6DoF relative arrangement of the RGB camera and the light emitter can be obtained. Note that, unlike detection via UWB, the detection results based on this third modification are asymmetrical between the devices.

[0348] 5-4. Fourth Revision

[0349] Next, the fourth modification will be described. Figure 58 This is an illustration of a collaborative display using a device including a face sensing unit A1 and a device without a face sensing unit A1. In the fourth modification, the device used for collaborative display does not need to include a face sensing unit A1.

[0350] like Figure 58 As shown, devices including the face sensing unit A1 are referred to as "Type-A". Devices without the face sensing unit A1 are referred to as "Type-B". Figure 58 In the examples, in devices 01 and 02 of type-B, the face detection status is always false in the status report. Alternatively, the corresponding processing can be skipped by assigning a specific value space to the device ID or by providing the device type separately to identify type-A / type-B. Type-B devices allow the omission of the camera block, enabling reasonable applications.

[0351] 5-5. Other modifications

[0352] In the processes described in the embodiments of this disclosure above, all or some of the processes described as automatically executed can be performed manually, or all or some of the processes described as manually executed can be automatically executed by known methods. Furthermore, unless otherwise stated, the processing procedures, specific names, and information including various data and parameters described in the foregoing documents and figures are freely changeable. For example, the various types of information shown in the figures are not limited to those shown.

[0353] The components of the device shown in the figure are functional concepts and are not necessarily physically configured as shown. That is to say, the specific implementation of the distribution and integration of the various devices is not limited to those shown in the figure, and all or part of each device can be functionally or physically distributed or integrated in any unit according to various loads, usage conditions, etc.

[0354] As long as the processed content does not contradict each other, the above-described embodiments of this disclosure can be appropriately combined. Furthermore, the order of steps shown in the sequence diagrams or flowcharts of this embodiment can be appropriately changed.

[0355] 6. Hardware Configuration

[0356] For example, the spatial display device 100 according to the above embodiments of this disclosure comprises having, for example, Figure 59 The computer 1000 with the configuration shown is implemented. Figure 59 This is a hardware configuration diagram illustrating an example of a computer 1000 that implements the functions of a spatial display device 100. The computer 1000 includes a central processing unit (CPU) 1100, random access memory (RAM) 1200, read-only memory (ROM) 1300, auxiliary storage device 1400, communication interface 1500, and input / output interface 1600. The units of the computer 1000 are connected together via a bus 1050.

[0357] The CPU 1100 operates based on programs stored in ROM 1300 or auxiliary storage device 1400 and controls each unit. For example, the CPU 1100 loads programs stored in ROM 1300 or auxiliary storage device 1400 into RAM 1200 and executes processing corresponding to various types of programs.

[0358] ROM 1300 stores boot programs such as the Basic Input / Output System (BIOS) executed by CPU 1100 when computer 1000 starts up, and programs that depend on the hardware of computer 1000.

[0359] The auxiliary storage device 1400 is a computer-readable recording medium that non-transitorily stores programs executed by the CPU 1100, data used by the programs, etc. Specifically, the auxiliary storage device 1400 is a recording medium that records programs according to this embodiment.

[0360] Communication interface 1500 is an interface used to connect computer 1000 to external network 1550. For example, CPU 1100 receives data from other devices or transmits data generated by CPU 1100 to other devices via communication interface 1500.

[0361] Input / output interface 1600 is an interface for connecting input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input devices such as a keyboard and mouse via input / output interface 1600. CPU 1100 also transmits data to output devices such as a monitor, speaker, and printer via input / output interface 1600. Additionally, input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium. Such media include, for example, optical storage media such as digital versatile discs (DVDs) or phase-change rewritable discs (PDs), magneto-optical storage media such as magneto-optical discs (MOs), magnetic tape media, magnetic storage media, or semiconductor memory.

[0362] For example, when computer 1000 is used as a spatial display device, the CPU 1100 of computer 1000 executes a program loaded onto RAM 1200 to implement the functions of control unit 101. Auxiliary storage device 1400 stores programs and data according to this disclosure. Note that CPU 1100 reads program data 1450 from auxiliary storage device 1400 and executes program data 1450. However, in another example, the CPU may obtain the program from another device via external network 1550.

[0363] In addition, not limited to reference Figure 59 The hardware configuration described herein. The functions implemented by the components described in this specification can be implemented in a circuit system or processing circuit system, including a general-purpose processor, a special-purpose processor, an integrated circuit, an application-specific integrated circuit (ASIC), a CPU, prior art circuits, and / or combinations thereof, which are programmed to implement the described functions. A processor includes transistors and other circuits and is considered a circuit system or processing circuit system. The processor can be a programmable processor that executes a program stored in memory. For example, the processor implements the functions of control unit 101 as a programmable processor. Note that the circuit system of the main display device according to this embodiment corresponds to the example of a "main circuit system," while the circuit system of the sub-display device corresponds to the example of a "sub-circuit system."

[0364] In this specification, circuit systems, units, and devices are hardware-programmed or hardware-executed to achieve the described functions. The hardware can be any hardware disclosed in this specification, or any known hardware that is programmed or otherwise executed to achieve the described functions.

[0365] In cases where the hardware is considered as a type of circuit system, the circuit system, device, or unit is a combination of hardware and software used to configure the hardware and / or processor.

[0366] 7. Conclusion

[0367] As described above, according to embodiments of this disclosure, the spatial display device 100 (corresponding to an example of "information processing device") includes a sensing unit 102 (corresponding to an example of "sensor"), a communication unit A5, and a control unit 101 (corresponding to an example of "circuit system"). The sensing unit 102 is configured to acquire a target device relative position D2 (corresponding to an example of "relative position information") of the main display device relative to the sub-display device. The communication unit A5 is configured to perform at least one of the following: transmitting viewpoint information of a user of the main display device to the sub-display device; or receiving a user's Face1 (corresponding to an example of "viewpoint information") or target device face sensing information D1 from the sub-display device. The control unit 101 is configured to acquire viewpoint information and relative position information, acquire device absolute position information D7 based on the relative position information indicating the position information of the main display device and the sub-display device in a coordinate system based on the main display device, and control the display of the main display device based on integrated viewpoint information D4 (corresponding to an example of "integrated information") or common information D10 or D10a based on the viewpoint information and device absolute position information D7. This enables collaborative control, including at least collaborative displays, through a simple array of multiple spatial display devices.

[0368] Although embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the embodiments described above, and various modifications can be made without departing from the spirit of the present disclosure. Furthermore, different embodiments and modified elements can be appropriately combined.

[0369] The effects of the embodiments described in this specification are merely illustrative and not limiting, and may provide other effects.

[0370] Note that this technology can also be configured as described below.

[0371] (1) An information processing device, comprising:

[0372] A sensor configured to acquire relative position information of the main display device relative to the sub-display device;

[0373] A communication unit is configured to perform at least one of the following: transmitting viewpoint information of a user from the main display device to the sub-display device, or receiving viewpoint information of the user from the sub-display device; and

[0374] The circuit system, which is configured as

[0375] Obtain the viewpoint information and the relative position information.

[0376] Based on the relative position information, obtain the absolute position information of the devices, which indicates the position information of the main display device and the sub-display device in the coordinate system based on the main display device, and...

[0377] The display of the main display device is controlled based on integrated information based on the viewpoint information and the absolute position information of the device.

[0378] (2) The information processing device according to (1), wherein,

[0379] The circuit system is configured as follows

[0380] The communication unit transmits a first request to the network connected to the main display device and the sub-display device, inquiring whether the main display device exists or not.

[0381] If no response to the first request is received after a specific time period has elapsed, the task of the main display device is executed.

[0382] (3) The information processing device according to (2), wherein,

[0383] The circuitry system serving as the main display device is configured to generate connection topology information related to the topology management of the main display device and the sub-display devices in the network, based on the viewpoint information and the device absolute position information.

[0384] (4) The information processing device according to (3), wherein,

[0385] The circuitry serving as the main display device is configured to transmit a response indicating that the main display device has been set as the transmission source of the first request via the communication unit and in accordance with the receipt of the first request via the communication unit.

[0386] (5) The information processing device according to (3) or (4), wherein,

[0387] The circuitry system serving as the main display device is configured to register the source of the second request as the sub-display device in the connection topology information based on the receipt of the second request via the communication unit, wherein the second request is a request for registration as the sub-display device.

[0388] (6) The information processing device according to (3), (4) or (5), wherein,

[0389] The circuitry serving as the main display device is configured to receive a report about the status of the sub-display device from the sub-display device via the communication unit, and to update the connection topology information based on the report.

[0390] (7) The information processing apparatus according to any one of (3) to (6), wherein,

[0391] The integrated information includes public information shared between the main display device and the sub-display device, and

[0392] The circuit system serving as the main display device is configured to...

[0393] Based on the viewpoint information, the device absolute position information, and the connection topology information, the public information is generated periodically or as needed.

[0394] The public information is transmitted via the communication unit to the sub-display device registered in the connection topology information.

[0395] (8) The information processing device according to (7), wherein,

[0396] The circuit system is configured to control the display of the 3D panel display unit of the sub-display device by transmitting the public information to the sub-display device.

[0397] (9) The information processing device according to (6), (7) or (8), wherein,

[0398] The connection topology information, for each of the main display device and the sub-display device, includes information related to the distance to the user and the confidence level of the acquisition of the viewpoint information, and

[0399] The circuit system is configured to select a new primary display device from the sub-display devices and the primary display device registered in the connection topology information based on a weighted average of the distance and the confidence level.

[0400] (10) The information processing apparatus according to any one of (1) to (9), wherein,

[0401] The integrated information includes viewpoint information acquired by each of the main display device and the sub-display device and shared between the main display device and the sub-display device, and

[0402] The circuit system is configured as follows

[0403] Based on the user's viewpoint information from the sub-display device, the main display device is controlled to display a viewpoint video with motion parallax based on the user's viewpoint from the sub-display device.

[0404] (11) The information processing device according to (10), wherein,

[0405] The viewpoint video includes autonomously manipulated virtual objects, and

[0406] The circuit system is configured as follows

[0407] The user's viewpoint information for the main display device is obtained from the sub-display device via the integrated information, and

[0408] Based on the viewpoint information of the user of the main display device, the main display device is controlled so that the virtual object performs an action according to at least one of the user's facial position or gaze direction.

[0409] (12) The information processing device according to (11), wherein,

[0410] The circuitry is configured to control the speakers of the main display device based on the viewpoint information of the user of the main display device, so as to output the audio of the virtual object according to the facial position or gaze direction of the user of the main display device.

[0411] (13) The information processing device according to (10), (11) or (12), wherein,

[0412] The viewpoint video includes autonomously manipulated virtual objects, and

[0413] The circuit system is configured as follows

[0414] The location information of the real object outside the detection range of the sensor included in the main display device is obtained from the sub-display device, and

[0415] Control the main display device so that the virtual object performs an action corresponding to the position information of the real object.

[0416] (14) The information processing device according to (13), wherein,

[0417] The circuit system is configured to control the speaker of the main display device to output the audio of the virtual object based on the position information of the real object.

[0418] (15) The information processing apparatus according to any one of (1) to (14), wherein,

[0419] The sensor includes a UWB sensor.

[0420] (16) An information processing device, comprising:

[0421] A sensor configured to acquire relative position information of the sub-display device relative to the main display device;

[0422] A communication unit configured to perform at least one of the following: transmitting viewpoint information of a user from the sub-display device to the main display device, or receiving viewpoint information of the user from the main display device; and

[0423] The circuit system, which is configured as

[0424] The viewpoint information and the relative position information are acquired and transmitted to the main display device via the communication unit.

[0425] The communication unit receives integrated information based on the device's absolute position information and the viewpoint information from the main display device. The absolute position information is obtained within the main display device based on the relative position information, and indicates the position information of the main display device and the sub-display device in the coordinate system based on the main display device.

[0426] The display of the sub-display device is controlled based on the integrated information.

[0427] (17) The information processing device according to (16), wherein,

[0428] The circuit system is configured as follows

[0429] The communication unit transmits a first request to the network connected to the main display device and the sub-display device, inquiring whether the main display device exists or not.

[0430] When a response to the first request is received within a specific time period, a second request is transmitted through the communication unit to the main display device indicated by the response. The second request is a request to register as the sub-display device, thereby receiving registration in the connection topology information related to the topology of the main display device and the sub-display device in the network.

[0431] (18) The information processing device according to (16) or (17), wherein,

[0432] The circuit system is configured as follows

[0433] The integrated information transmitted from the main display device is received periodically or as needed through the communication unit.

[0434] When the integration information includes identification information indicating the main display device, the task is performed as the main display device, rather than as the sub-display device.

[0435] (19) An information processing system, comprising:

[0436] Main display device; and

[0437] Sub-display devices, among which,

[0438] The main display device includes

[0439] A sensor is configured to acquire relative position information of the main display device relative to the sub-display device.

[0440] A communication unit configured to perform at least one of the following: transmitting viewpoint information of a user from the main display device to the sub-display device, or receiving viewpoint information of the user from the sub-display device; and

[0441] The main circuit system is configured as follows:

[0442] Obtain the viewpoint information and the relative position information.

[0443] Based on the relative position information, obtain the absolute position information of the devices, which indicates the position information of the main display device and the sub-display device in the coordinate system based on the main display device, and...

[0444] The display of the main display device is controlled based on integrated information based on the viewpoint information and the absolute position information of the device.

[0445] The sub-display device includes

[0446] A sensor is configured to acquire relative position information of the sub-display device relative to the main display device.

[0447] A communication unit configured to perform at least one of the following: transmitting viewpoint information of a user from the sub-display device to the main display device, or receiving viewpoint information of the user from the main display device; and

[0448] Sub-circuit system, which is configured as

[0449] The viewpoint information and the relative position information are acquired and transmitted to the main display device via the communication unit.

[0450] The communication unit receives integrated information based on the viewpoint information and the absolute device position information, obtained from the main display device based on the relative position information.

[0451] The display of the sub-display device is controlled based on the integrated information.

[0452] (20) An information processing method performed by an information processing device, the information processing device comprising a sensor and a communication unit, the sensor being configured to acquire relative position information of a main display device relative to a sub-display device, the communication unit being configured to perform at least one of the following: transmitting viewpoint information of a user of the main display device to the sub-display device, or receiving viewpoint information of the user from the sub-display device, the information processing method comprising:

[0453] Obtain the viewpoint information and the relative position information;

[0454] Based on the relative position information, obtain absolute device position information indicating the position information of the main display device and the sub-display device in the coordinate system based on the main display device; and

[0455] The display of the main display device is controlled based on integrated information based on the viewpoint information and the absolute position information of the device.

[0456] List of reference numerals

[0457] 1 Display System

[0458] 100 spatial display devices

[0459] 101 Control Unit

[0460] 102 sensing units

[0461] 1033D panel display unit

[0462] 200 wireless router

[0463] A1 Face Sensing Unit

[0464] A2 Relative Position Sensing Unit

[0465] A3 Information Processing Unit

[0466] A4 Video Processing Unit

[0467] A5 Communication Unit

[0468] C1 Face Sensing Unit

[0469] C10 from device information receiving unit

[0470] C11 Public Information Transmission Unit

[0471] C12 Public Information Analysis Unit

[0472] C13 from device information generation unit

[0473] C14 Device Information Transmission Unit

[0474] C15 Public Information Receiving Unit

[0475] C2 Relative Position Sensing Unit

[0476] C3 connects to the topology control unit.

[0477] C4 Integrated Viewpoint Information Generation Unit

[0478] C5 Equipment Position Coordinate Transformation Unit

[0479] C6 Public Information Generation Unit

[0480] C7 Video Generation Unit

[0481] C8 from the device information analysis unit

[0482] C9 Video Display Unit

[0483] D1 Target Device Facial Sensing Information

[0484] D10 Public Information

[0485] D10a Public Information

[0486] D2 Relative position of target equipment

[0487] D3 connection topology information

[0488] D4 Integrated Viewpoint Information

[0489] D5 other device facial sensing information

[0490] Relative positions of other equipment in D6

[0491] D7 device absolute position information

[0492] D8 Panel Video

[0493] D9 from device information

Claims

1. An information processing device, comprising: A sensor configured to acquire relative position information of the main display device relative to the sub-display device; A communication unit configured to perform at least one of the following: transmitting viewpoint information of a user from the main display device to the sub-display device, or receiving viewpoint information of the user from the sub-display device; as well as The circuit system, which is configured as Obtain the viewpoint information and the relative position information. Based on the relative position information, obtain the absolute position information of the devices, which indicates the position information of the main display device and the sub-display device in the coordinate system based on the main display device, and... The display of the main display device is controlled based on integrated information based on the viewpoint information and the absolute position information of the device.

2. The information processing device according to claim 1, wherein, The circuit system is configured as follows The communication unit transmits a first request to the network connected to the main display device and the sub-display device, inquiring whether the main display device exists or not. If no response to the first request is received after a specific time period has elapsed, the task of the main display device is executed.

3. The information processing device according to claim 2, wherein, The circuitry system serving as the main display device is configured to generate connection topology information related to the topology management of the main display device and the sub-display devices in the network, based on the viewpoint information and the device absolute position information.

4. The information processing device according to claim 3, wherein, The circuitry serving as the main display device is configured to transmit a response indicating that the main display device has been set as the transmission source of the first request via the communication unit and in accordance with the receipt of the first request via the communication unit.

5. The information processing device according to claim 3, wherein, The circuitry system serving as the main display device is configured to register the source of the second request as the sub-display device in the connection topology information based on the receipt of the second request via the communication unit, wherein the second request is a request for registration as the sub-display device.

6. The information processing device according to claim 3, wherein, The circuitry serving as the main display device is configured to receive a report about the status of the sub-display device from the sub-display device via the communication unit, and to update the connection topology information based on the report.

7. The information processing device according to claim 3, wherein, The integrated information includes public information shared between the main display device and the sub-display device, and The circuit system serving as the main display device is configured to... Based on the viewpoint information, the device absolute position information, and the connection topology information, the public information is generated periodically or as needed. The public information is transmitted via the communication unit to the sub-display device registered in the connection topology information.

8. The information processing device according to claim 7, wherein, The circuit system is configured to control the display of the 3D panel display unit of the sub-display device by transmitting the public information to the sub-display device.

9. The information processing device according to claim 6, wherein, The connection topology information, for each of the main display device and the sub-display device, includes information related to the distance to the user and the confidence level of the acquisition of the viewpoint information, and The circuit system is configured to select a new primary display device from the sub-display devices and the primary display device registered in the connection topology information based on a weighted average of the distance and the confidence level.

10. The information processing device according to claim 1, wherein, The integrated information includes viewpoint information acquired by each of the main display device and the sub-display device and shared between the main display device and the sub-display device, and The circuit system is configured as follows Based on the user's viewpoint information from the sub-display device, the main display device is controlled to display a viewpoint video with motion parallax based on the user's viewpoint from the sub-display device.

11. The information processing device according to claim 10, wherein, The viewpoint video includes autonomously manipulated virtual objects, and The circuit system is configured as follows The user's viewpoint information for the main display device is obtained from the sub-display device via the integrated information, and Based on the viewpoint information of the user of the main display device, the main display device is controlled so that the virtual object performs an action according to at least one of the user's facial position or gaze direction.

12. The information processing device according to claim 11, wherein, The circuitry is configured to control the speakers of the main display device based on the viewpoint information of the user of the main display device, so as to output the audio of the virtual object according to the facial position or gaze direction of the user of the main display device.

13. The information processing device according to claim 10, wherein, The viewpoint video includes autonomously manipulated virtual objects, and The circuit system is configured as follows The location information of the real object outside the detection range of the sensor included in the main display device is obtained from the sub-display device, and Control the main display device so that the virtual object performs an action corresponding to the position information of the real object.

14. The information processing device according to claim 13, wherein, The circuit system is configured to control the speaker of the main display device to output the audio of the virtual object based on the position information of the real object.

15. The information processing device according to claim 1, wherein, The sensor includes a UWB sensor.

16. An information processing device, comprising: A sensor configured to acquire relative position information of the sub-display device relative to the main display device; A communication unit configured to perform at least one of the following: transmitting viewpoint information of a user of the sub-display device to the main display device, or receiving viewpoint information of the user from the main display device; as well as The circuit system, which is configured as The viewpoint information and the relative position information are acquired and transmitted to the main display device via the communication unit. The communication unit receives integrated information based on the device's absolute position information and the viewpoint information from the main display device. The absolute position information is obtained within the main display device based on the relative position information, and indicates the position information of the main display device and the sub-display device in the coordinate system based on the main display device. The display of the sub-display device is controlled based on the integrated information.

17. The information processing device according to claim 16, wherein, The circuit system is configured as follows The communication unit transmits a first request to the network connected to the main display device and the sub-display device, inquiring whether the main display device exists or not. When a response to the first request is received within a specific time period, a second request is transmitted through the communication unit to the main display device indicated by the response. The second request is a request to register as the sub-display device, thereby receiving registration in the connection topology information related to the topology of the main display device and the sub-display device in the network.

18. The information processing device according to claim 16, wherein, The circuit system is configured as follows The integrated information transmitted from the main display device is received periodically or as needed through the communication unit. When the integration information includes identification information indicating the main display device, the task is performed as the main display device, rather than as the sub-display device.

19. An information processing system, comprising: Main display device; as well as Sub-display devices, among which, The main display device includes A sensor is configured to acquire relative position information of the main display device relative to the sub-display device. A communication unit configured to perform at least one of the following: transmitting viewpoint information of a user from the main display device to the sub-display device, or receiving viewpoint information of the user from the sub-display device; and The main circuit system is configured as follows: Obtain the viewpoint information and the relative position information. Based on the relative position information, obtain the absolute position information of the devices, which indicates the position information of the main display device and the sub-display device in the coordinate system based on the main display device, and... The display of the main display device is controlled based on integrated information based on the viewpoint information and the absolute position information of the device. The sub-display device includes A sensor is configured to acquire relative position information of the sub-display device relative to the main display device. A communication unit configured to perform at least one of the following: transmitting viewpoint information of a user from the sub-display device to the main display device, or receiving viewpoint information of the user from the main display device; and Sub-circuit system, which is configured as The viewpoint information and the relative position information are acquired and transmitted to the main display device via the communication unit. The communication unit receives integrated information based on the viewpoint information and the absolute device position information, obtained from the main display device based on the relative position information. The display of the sub-display device is controlled based on the integrated information.

20. An information processing method executed by an information processing device, the information processing device comprising a sensor and a communication unit, the sensor being configured to acquire relative position information of a main display device relative to a sub-display device, the communication unit being configured to perform at least one of the following: transmitting viewpoint information of a user from the main display device to the sub-display device, or receiving viewpoint information of a user from the sub-display device, the information processing method comprising: Obtain the viewpoint information and the relative position information; Based on the relative position information, obtain the absolute position information of the device that indicates the position information of the main display device and the sub-display device in the coordinate system based on the main display device; as well as The display of the main display device is controlled based on integrated information based on the viewpoint information and the absolute position information of the device.

Images