Video conferencing compartment

Abstract

A video conferencing compartment (100) and a method of operation thereof. The video conferencing compartment (100) includes a tracking system (106), a stereoscopic projector (104), a first actuator (114) configured to translate the stereoscopic projector (104), an image sensor device (108), a second actuator (118) configured to translate the image sensor device (108), and a controller (202). The controller (202) is configured to obtain a first stream of the first location from the tracking system (106), transmit a command to the first actuator (114) to adjust a location of the stereoscopic projector (104) based on the first stream of the first location, and transmit the first stream of the first location to a remote video conferencing compartment. The controller (202) is further configured to receive a second stream of a second location from the remote video conferencing compartment and transmit a command to the second actuator (118) to adjust a location of the image sensor device (108) based on the second stream of the second location.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 345,127 filed on May 24, 2022, U.S. Provisional Patent Application No. 63 / 421,854 filed on November 2, 2022, and European Patent Application No. 22175095.3 filed on May 24, 2022, each of which is incorporated herein by reference in its entirety. Background of the Invention 1. Technical Field

[0003] The present application relates generally to systems and methods for video conferencing. 2. Background technology

[0004] Virtual meetings (such as video conferences) are usually conducted in personal offices and conference rooms, allowing participants to conduct virtual meetings from different locations. The conference system includes, for example, a display, speakers, cameras, and microphones, which allow participants to see and communicate with other participants. Participants in the virtual meeting see other participants via the display.

[0005] GB 2 353 429 A discloses an arrangement for displaying a life-size, real-time image of a person from a remote location in a three-dimensional setting in a home location, while providing the person at the remote location with a corresponding remote presentation of the home location. The arrangement comprises: a video presentation system for displaying the person on a black or chromakey background; a two-way reflector for viewing a superimposed video image of the environment and the person; a video camera or a pair of cameras positioned in line with the eyes of the superimposed image of the person; and a network connection between the home location and the remote location. Stereoscopic viewing may be used with the two-way reflector and a retroreflective surface to display a life-size, autostereoscopic, real-time image of the person from the remote location in a 3D setting. Further arrangements include a device that allows tracking of the user's eyes. Summary of the invention

[0006] While virtual meetings provide a face-to-face meeting experience, the displays used in traditional virtual meeting settings only provide two-dimensional imaging. As a result, participants in virtual meetings cannot sense the presence of other participants and may still experience social disconnection from other participants in the virtual meeting.

[0007] Embodiments described herein relate to a video conferencing booth. The video conferencing booth is designed to provide a sense of a face-to-face meeting, avoid the need for glasses to provide three-dimensional images, and reduce or minimize reflections and obstructions of the screen. In addition, three-dimensional systems with full parallax typically require complex three-dimensional scene sensing, image compression algorithms, and image reconstruction algorithms, which require a lot of computing and bandwidth requirements. Embodiments described herein provide a three-dimensional video conferencing booth that can use mechanical components (such as actuators) to match views between participants. Because the components are mechanically moved based on tracked head and / or facial movements, the system can use only stereoscopic video streams, and the amount of metadata required for calculation is relatively small.

[0008] In one exemplary aspect of the present disclosure, a video conferencing booth is provided, the video conferencing booth comprising: a tracking system configured to track the position of a first user, a stereo projector; a first actuator configured to translate the stereo projector along at least a first axis; an image sensor device comprising a first image sensor configured to generate a video stream corresponding to a left eye of a second user of a remote video conferencing booth, and a second image sensor configured to generate a video stream corresponding to a right eye of the second user of the remote video conferencing booth; a second actuator configured to translate the image sensor device along at least a second axis; and a controller configured to: obtain a first stream of a first position of the first user from the tracking system, transmit a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first position of the first user, and transmit the first stream of the first position of the first user to the remote video conferencing booth. , receiving from the remote video conferencing booth a second stream of a second position of the second user of the remote video conferencing booth, transmitting a command to the second actuator to adjust the position of the image sensor device along the second axis based on the second stream of the second position of the second user, receiving from the remote video conferencing booth a video stream corresponding to a left eye of the second user and a video stream corresponding to a right eye of the second user, wherein each of the received video streams of the second user includes at least a given frame captured based on the first user being at a third position, determining that the first user is at a fourth position when receiving the given frame of the video streams based at least in part on the first stream of the first position of the first user from the tracking system, estimating an error between the third position of the first user and the fourth position of the first user, and adjusting the video stream corresponding to the left eye of the second user and the video stream corresponding to the right eye of the second user based on the estimated error.

[0009] In another exemplary aspect of the present disclosure, a method for operating a video conferencing booth is provided, the video conferencing booth comprising: a tracking system configured to track the position of a first user; a stereo projector; a first actuator configured to translate the stereo projector along at least a first axis; an image sensor device comprising a first image sensor configured to generate a video stream corresponding to a left eye of a second user of the remote video conferencing booth, and a second image sensor configured to generate a video stream corresponding to a right eye of the second user of the remote video conferencing booth; and a second actuator configured to translate the image sensor device along at least a second axis, the method comprising: obtaining a first stream of a first position of the first user from the tracking system, transmitting a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first position of the first user, and transmitting the first stream of the first position of the first user to the remote video a conference booth, receiving from the remote video conference booth a second stream of a second position of the second user of the remote video conference booth, transmitting a command to the second actuator to adjust the position of the image sensor device along the second axis based on the second stream of the second position of the second user, receiving from the remote video conference booth a video stream corresponding to a left eye of the second user and a video stream corresponding to a right eye of the second user, wherein each of the received video streams of the second user includes at least a given frame captured based on the first user being at a third position, determining that the first user is at a fourth position when receiving the given frame of the video streams based at least in part on the first stream of the first position of the first user from the tracking system, estimating an error between the third position of the first user and the fourth position of the first user, and adjusting the video stream corresponding to the left eye of the second user and the video stream corresponding to the right eye of the second user based on the estimated error.

[0010] In another exemplary aspect of the present disclosure, a non-transitory computer-readable medium storing instructions is provided, which, when executed by a processor of a projection system in a video conferencing booth according to the present disclosure, causes the projection system to perform operations, the operations including the following: obtaining a first stream of a first position of the first user from the tracking system, transmitting a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first position of the first user, transmitting the first stream of the first position of the first user to a remote video conferencing booth, receiving a second stream of a second position of a second user of the remote video conferencing booth from the remote video conferencing booth, and transmitting a command to the second actuator to adjust the position of the image sensor device along the second axis based on the second stream of the second position of the second user.

[0011] In another exemplary aspect of the present disclosure, a video conference table is provided, the video conference table comprising: a tracking system configured to track the positions of a first local user and a second local user; a first image sensor device configured to capture at least the first local user; a second image sensor device configured to capture at least the second local user; a first actuator configured to translate the first image sensor device along at least a first axis; a second actuator configured to translate the second image sensor device along at least a second axis; and a controller. The controller is configured to receive a first stream of a first position of a first remote user, and transmit a command to the first actuator to adjust the position of the first image sensor device along the first axis based on the first stream of the first position of the first remote user. The controller is configured to receive a second stream of a second position of a second remote user, and transmit a command to the second actuator to adjust the position of the second image sensor device along the second axis based on the second stream of the second position of the second remote user. The first image sensor device comprises a first image sensor of the first image sensor device and a second image sensor of the first image sensor device. The first image sensor of the first image sensor device is configured to generate a video stream corresponding to a left eye of a user of a remote video conferencing booth, and the second image sensor of the first image sensor device is configured to generate a video stream corresponding to a right eye of the user of the remote video conferencing booth. The second image sensor device includes a first image sensor of the second image sensor device and a second image sensor of the second image sensor device. The first image sensor of the second image sensor device is configured to generate a video stream corresponding to a left eye of the user of the remote video conferencing booth, and the second image sensor of the second image sensor device is configured to generate a video stream corresponding to a right eye of the user of the remote video conferencing booth.

[0012] In another exemplary aspect of the present disclosure, a method for operating a video conference table is provided, the video conference table comprising: a tracking system configured to track the positions of a first local user and a second local user; a first image sensor device configured to capture at least the first local user; a second image sensor device configured to capture at least the second local user; a first actuator configured to translate the first image sensor device along at least a first axis; and a second actuator configured to translate the second image sensor device along at least a second axis. The method comprises receiving a first stream of a first position of a first remote user, and transmitting a command to the first actuator to adjust the position of the first image sensor device along the first axis based on the first stream of the first position of the first remote user. The method comprises receiving a second stream of a second position of a second remote user, and transmitting a command to the second actuator to adjust the position of the second image sensor device along the second axis based on the second stream of the second position of the second remote user. The first image sensor device comprises a first image sensor of the first image sensor device and a second image sensor of the first image sensor device. The first image sensor of the first image sensor device is configured to generate a video stream corresponding to a left eye of a user of a remote video conferencing booth, and the second image sensor of the first image sensor device is configured to generate a video stream corresponding to a right eye of the user of the remote video conferencing booth. The second image sensor device includes a first image sensor of the second image sensor device and a second image sensor of the second image sensor device. The first image sensor of the second image sensor device is configured to generate a video stream corresponding to a left eye of the user of the remote video conferencing booth, and the second image sensor of the second image sensor device is configured to generate a video stream corresponding to a right eye of the user of the remote video conferencing booth.

[0013] In another exemplary aspect of the present disclosure, a video conference booth is provided, the video conference booth comprising: a tracking system configured to track the position of a first user; a wearable stereo projector; an image sensor device comprising a first image sensor configured to generate a video stream corresponding to a left eye of a second user of the remote video conference booth, and a second image sensor configured to generate a video stream corresponding to a right eye of the second user of the remote video conference booth; an actuator configured to translate the image sensor device along at least a first axis; and a controller. The controller is configured to obtain a first stream of a first position of the first user from the tracking system, transmit the first stream of the first position of the first user to the remote video conference booth, receive a second stream of a second position of the second user of the remote video conference booth from the remote video conference booth, and transmit a command to the actuator to adjust the position of the image sensor device along the first axis based on the second stream of the second position of the second user.

[0014] In this way, various aspects of the present disclosure provide for capturing and displaying images and motion tracking, and provide effective improvements in at least the technical fields of image projection, image display, motion tracking, three-dimensional imaging, and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] These and other more detailed and specific features of various embodiments are more fully disclosed in the following description with reference to the accompanying drawings, in which:

[0016] Figure 1 An example video conferencing booth is depicted.

[0017] Figure 2 Describes the use Figure 1 Example control system for a video conferencing booth.

[0018] Figure 3 Depicted is an example camera.

[0019] Figure 4 Depicted by Figure 2 An example method for performing a control system.

[0020] Figure 5 Depicted by Figure 2 Another example method performed by a control system.

[0021] Figure 6 Another example video conferencing booth is depicted.

[0022] Figure 7 Depicted by Figure 2 Another example method performed by a control system.

[0023] FIG. 8A to FIG. 8B An example video conferencing table is depicted. DETAILED DESCRIPTION

[0024] The present disclosure and its various aspects may be embodied in various forms, including: hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​the various aspects of the present disclosure and is not intended to limit the scope of the present disclosure in any way.

[0025] In the following description, many details such as optical device configuration, timing, operation, etc. are set forth to provide an understanding of one or more aspects of the present disclosure. It is obvious to those skilled in the art that these specific details are only exemplary and are not intended to limit the scope of the present application.

[0026] Video conference booth configuration

[0027] Figure 1 An example video conferencing booth (100) is illustrated. A user (102) (e.g., a first user) is within the video conferencing booth (100). The video conferencing booth (100) includes a projector (104), a tracking system (106), and an image sensor device (108) (e.g., a camera). Figure 1 In the embodiment of the present invention, the projector (104) is a stereoscopic projector that projects a first image (130) and a second image (132), each image corresponding to an eye of the user (102). For example, the first image (130) may correspond to a left eye image, and the second image (132) may correspond to a right eye image. The stereoscopic projection provides a three-dimensional (3D) viewing experience for the user (102). In some embodiments, the projector (104) is a micro projector. In some embodiments, the image sensor device (108) is a high dynamic range (HDR) camera, and the projector (104) is an HDR projector. Additionally, in some cases, two separate projectors may be implemented, rather than a single projector (104) projecting the first image (130) and the second image (132) from the same projector housing. Thus, the first projector projects the first image (130), and the second projector projects the second image (132).

[0028] exist Figure 1In the example shown, the projector (104) projects a first image (130) and a second image (132) toward a reflector (120). The reflector (120) reflects the first image (130) and the second image (132) toward a splitter (122) (e.g., a beam splitter). In some embodiments, the splitter (122) is a polarization splitter (122) (e.g., a polarization beam splitter) that reflects light waves of a specific polarization while blocking (or allowing light of other polarizations to pass through). The light of the first image (130) and the second image (132) reflected by the splitter (122) is reflected toward a screen (124). In some configurations, the splitter (122) is located directly between the user (102) and the screen (124). Therefore, in order to allow the user (102) to view the first image (130) and the second image (132) on the screen (124) in this configuration, the splitter (122) can be a bidirectional splitter (122). In this manner, the user (102) does not notice the presence of the splitter (122). In some embodiments, the first image (130) and the second image (132) are projected directly onto the splitter (122) rather than including the reflector (120). Additionally, in some cases, the polarizing filter is optically located downstream of the projector (104) such that the splitter (122) reflects light from the projector (104). The polarizing filter may be integrated into the projector (104) or may be part of the projector.

[0029] By projecting a first image (130) and a second image (132) onto a screen (124), the screen (124) provides a stereoscopic image. This provides a seemingly three-dimensional experience to the user (102) because the image viewed on the screen (124) provides an impression of depth. In some embodiments, the screen (124) is a retroreflective screen (124). Thus, the screen (124) can reflect light back in the direction in which the light was projected onto the screen (124) (i.e., toward the splitter (122)). The splitter (122) allows the light reflected by the screen (124) to pass through toward the user (102). Thus, the first image (130) is provided to the right eye of the user (102) and the second image (132) is provided to the left eye of the user (102). Because the first image (130) and the second image (132) are provided directly to the user (102) and perpendicular to the screen (124), the use of polarization or spectrally selective eyewear that may be typically required in a 3D viewing system is avoided. In some embodiments, the screen (124) includes a quarter wave plate configured to "twist" polarized light from the splitter (122) and reduce secondary reflections. Because the light is twisted by the quarter wave plate, the polarized light is able to pass through the splitter (122) to the user (102). More specifically, in an embodiment including a quarter wave plate and a polarizing filter, a polarizing filter implemented in front of or within the projector (104) applies a first polarization so that the light is reflected off the splitter (122). The quarter wave plate then "twists" the light into a second polarization so that the light passes through the splitter (122) and is emitted toward the user (102).

[0030] When the user (102) is participating in the video conference and viewing the screen (124), the image sensor device (108) captures an image or video of the user (102), which is provided to the remote video conference booth, as discussed in more detail below. In some embodiments, the image sensor device (108) directly captures the user (102). In some embodiments, the image sensor device (108) receives light (e.g., reflection) of the user (102) from the splitter (122). Although the image sensor device (108) is shown as a stereo image sensor device (108) providing two image sensor devices in the same housing, the video conference booth (100) may include additional or separate image sensors, such as an image sensor device (108) including a pair of image sensors having separate housings. Each image sensor in the pair of image sensors of the image sensor device (108) may correspond to an eye of the remote user. For example, the first image sensor device generates a video stream corresponding to the left eye, and the second image sensor device generates a video stream corresponding to the right eye. In some cases, a polarizer is attached to (or integrated into) the lens of each image sensor of the image sensor device (108) to reduce residual transmission of the projector (104).

[0031] In addition, the tracking system (106) tracks the head and facial movements of the user (102). In some cases, the tracking system (106) is an infrared or near infrared tracking system (106). In this case, a light source can be implemented and projected onto the user (102) to assist in depth determination. In one example, the infrared tracking system (106) uses a marker (such as a reflective marker coupled to the user (102)) to track the movement of the user (102). In another example, the tracking system (106) is configured to track the movement of the user (102) using an eye tracking algorithm, a facial tracking algorithm, and other algorithms that may be machine learning algorithms. The image sensor device (108), the tracking system (106), and the projector (104) can each be coupled to a sensor actuator (118), a tracking actuator (116), and a projector actuator (114), as described in more detail below.

[0032] In some embodiments, the video conferencing booth (100) further includes a microphone and a speaker (e.g., a spatial audio device) for transmitting audio data. For example, the microphone within the video conferencing booth (100) captures audio provided by the user (102) and converts the audio into data for transmission to the remote video conferencing booth. Additionally, during the video conference, the speaker outputs data received from the remote video conferencing booth. In order to reproduce the sound so that it sounds like it is emanating from the location of the remote participant (e.g., from the remote participant himself), the speaker can be implemented within the image sensor device (108) (in Figure 1In some cases, the speakers may be implemented behind the screen (124). In this case, the screen (124) is acoustically transparent.

[0033] In some embodiments, the user (102) wears a wearable stereo projector in place of the projector (104). For example, the wearable stereo projector may be glasses or eyeglass frames, including small projectors embedded in the left and right sides of the eyeglass frames. The wearable stereo projector projects a left eye image and a right eye image onto a retroreflective screen (124). The screen (124) reflects the left eye image into the left eye of the user (102) and reflects the right eye image into the right eye of the user (104). Although the wearable stereo projector may be configured as glasses, the glasses are not necessarily polarized or spectrally selective eyewear. Instead, the glasses may be simply eyeglass frames (e.g., without any glass or lenses). In further embodiments, the user (102) may wear augmented reality (AR) glasses (instead of the projector (104) and the wearable stereo projector), the glasses being configured to provide a three-dimensional image of a remote participant. In such embodiments, components such as the reflector (120) and the projector (104) and associated components may be omitted.

[0034] Video conferences usually involve multiple participants. Therefore, the video conference booth (100) can transmit audio data, video data and operational data to a second remote video conference booth during the video conference. Figure 2 An example video conferencing system (200) for controlling a plurality of video conferencing booths (100) is provided. Figure 2 The example provides a local video conference booth controller (202) (e.g., a local controller) and a remote video conference booth controller (252) (e.g., a remote controller), each controller controlling a local video conference booth and a remote video conference booth, respectively. However, in other cases, additional video conference booths may be included. The local controller (202) and the remote controller (252) include substantially similar components.

[0035] The local controller (202) includes an electronic processor (204), a memory (206), and a transceiver (214), etc. The electronic processor (204), the memory (206), and the transceiver (214) communicate via one or more control and / or data buses. Figure 2Only one example of a local controller (202) is illustrated. The local controller (202) may include more or fewer components and may perform functions in addition to those explicitly described herein. In some examples, the electronic processor (204) is implemented as a microprocessor with a separate memory (206). In other examples, the electronic processor (204) is implemented as a microcontroller with the memory (206) on the same chip. The electronic processor (204) may be implemented with multiple processors and may be implemented in part or in whole, for example, as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0036] The memory (206) includes non-transitory computer-readable memory that stores instructions received and executed by the electronic processor (204) to perform the functions of the video conferencing booth (100) described herein. The memory (206) may include, for example, a combination of different types of memory, such as read-only memory and random access memory.

[0037] The transceiver (214) allows the local controller (202) to perform wired and / or wireless communication with the remote controller (252) via the network (230). The network (230) may be, for example, a Long Term Evolution (LTE) network, a Bluetooth TM The transceiver (214) may also handle communications with various components and devices of the video conferencing booth (100) connected to the local controller (202), such as the projector (104), tracking system (106), image sensor device (108), microphone (210), speaker (140), projector actuator (114), and sensor actuator (118).

[0038] The local controller (202) receives and transmits data from connected components operating in the video conference booth (100). For example, the local controller (202) receives images and / or video of the user (102) from the image sensor device (108). The local controller (202) receives tracking information about the head and / or facial movements of the user (102) from the tracking system (106). In addition, the local controller (202) receives audio data from the microphone (210), including voice and other audio present in the video conference booth (100). The video, tracking information, and audio data of the user (102) are provided to the remote controller (252) via the network (230). The tracking information associated with the user (102) can also be used to control the projector actuator (114), as described in more detail below.

[0039] The local controller (202) receives similar image and / or video data, tracking information, and audio data associated with a remote user (not shown) from the remote controller (252). The received audio data (212) is output using a speaker. The image and / or video data is output using a projector (104). Tracking information associated with the remote user can be used to control a sensor actuator (118), as described in more detail below.

[0040] How to operate a video conference booth

[0041] To enhance the stereoscopic experience of the user (102), the local controller (202) controls the projector actuator (114) and the sensor actuator (118) to move the projector (104) and the image sensor device (108), respectively. For example, Figure 3 A projector (104) coupled to a projector actuator (114) is illustrated. The projector actuator (114) is configured to move the projector (104) along at least one of a first axis (X) (e.g., an X-axis), a second axis (Y) (e.g., a Y-axis), and a third axis (Z) (e.g., a Z-axis). In some embodiments, the projector actuator (114) is further configured to twist and rotate the projector (104) along at least one of a fourth axis (θ1), a fifth axis (θ2), and a sixth axis (θ3). Although only the projector (104) is shown, the image sensor (108) can be configured to move in the same manner via the sensor actuator (118). The actuator can be, for example, a robotic arm configured to move with several degrees of freedom (e.g., up to 6 degrees of freedom). The selection of axes for translating and / or rotating the projector (104) and / or the image sensor device (108) depends on (i) the quality of the stereoscopic experience selected to enhance the user (102), and (ii) the axes along which the user (102) has a certain degree of freedom of movement. Although the best stereoscopic experience can be expected when translating along three axes and rotating around another three axes, it may be reasonable to limit the axes of translation and / or rotation to those axes along which the user (102) to be tracked has sufficient freedom of movement. For example, in a setting where the user (102) is sitting in a chair and the user's movement is limited to movement of the head mainly along a horizontal axis, it may even be sufficient to provide translation of the projector and image sensor device along the horizontal axis. Obviously, any additional axes provided for translating and rotating the projector and image sensor device will improve the stereoscopic experience. However, whether to provide translation along a specific axis or rotation around a specific axis where the user has only limited freedom of movement is a trade-off between improving quality and reducing complexity. Therefore, the present invention can be implemented using various numbers of axes of translation and / or rotation.

[0042] In order to maintain a clear stereoscopic three-dimensional image as the user (102) moves his head and face, the local controller (202) adjusts the position of the projector (104) based on the tracked movements provided by the tracking system (106). By adjusting the position of the projector (104) in this manner, the first image (130) and the second image (132) can be provided to the left and right eyes of the user (102) even when the user (102) moves around. Figure 4 A method (400) for controlling components of a video conferencing booth (100) based on the movement of a user (102) is provided. The method (400) can be performed, for example, by a local controller (202) or a remote controller (252). In addition, Figure 4 The steps provided in are merely examples and may be performed in a different order or simultaneously.

[0043] At step (402), the local controller (202) obtains a first position stream from the tracking system (106). For example, when the user (102) speaks or otherwise moves his head and face during a video conference, the tracking system (106) monitors the movement of the local user (102). The tracking system (106) then transmits the movement position of the user (102) to the local controller (202).

[0044] At step (404), the local controller (202) obtains a first video stream from the image sensor device (108). For example, when the user (102) is having a video conference, the image sensor device (108) captures the user (102) through pictures and / or videos. The image sensor device (108) provides the captured images and / or videos to the local controller (202).

[0045] At step (406), the local controller (202) controls the first actuator to adjust the projector (104). For example, as the head position of the local user (102) changes, the first image (130) and the second image (132) may no longer be aligned with the eyes of the user (102) when reflected by the screen (124). Therefore, in order to maintain proper eye contact and ensure that the video quality remains consistent, the local controller (202) controls the projector actuator (114) to adjust the position of the projector (104). The projector actuator (114) can move the projector (104) along the first axis (X) or the second axis (Y) or the third axis (Z) or the fourth axis (θ1) or the fifth axis (θ2) or the sixth axis (θ3) or a combination thereof. Depending on the quality selected for enhancing the stereoscopic experience, the projector actuator (114) can be limited to a subset of the above six axes.

[0046] At step (408), the local controller (202) transmits the first position stream and the first video stream to the remote controller (252) (i.e., the remote video conference booth). In this way, the remote controller (252) can provide the first video stream on the corresponding remote screen. In addition, the remote controller (252) can control the connected components according to the first position stream.

[0047] When the local controller (202) transmits the position and video to the remote controller (252), the local controller (202) also receives the position and video of the corresponding remote user using the remote video conference booth from the remote controller (252). Figure 5 A method (500) for controlling components of a video conferencing booth (100) based on data associated with a remote participant is provided. The method (500) may be performed, for example, by a local controller (202) or a remote controller (252). In addition, Figure 5 The steps provided in are merely examples and may be performed in a different order or simultaneously.

[0048] At step (502), the local controller (202) receives a second position stream from the remote video conference booth. For example, when a remote user in the remote video conference booth speaks or otherwise moves his head and face during a video conference, a tracking system (106) associated with the remote controller (252) monitors the movement. The tracking system (106) provides the remote controller (252) with the movement of the remote user. The remote controller (252) then transmits the movement of the remote user to the local controller (202) via the network (230).

[0049] At step (504), the local controller (202) receives a second video stream from the remote video conference booth. For example, when a remote user participates in a video conference, an image sensor device (108) associated with the remote controller (252) captures the remote user through pictures and / or videos. The image sensor device (108) provides the captured images and / or videos to the remote controller (252). The remote controller (252) then transmits the captured images and / or videos of the remote user to the local controller (202) via the network (230). At step (506), the local controller (202) displays the second video stream. For example, the image and / or video of the remote user is provided on the screen (124) using the projector (104).

[0050] At step (508), the local controller (202) controls the second actuator to adjust the image sensor device (108). For example, when the remote user's head position changes, the local controller (202) controls the sensor actuator (118) to adjust the position of the image sensor device (108). The sensor actuator (118) can move the image sensor device (108) along a first axis (X), a second axis (Y), a third axis (Z), a fourth axis (θ1), a fifth axis (θ2), a sixth axis (θ3), or a combination thereof.

[0051] Due to network latency, there may be a delay in the movement of the user (102) and the remote user relative to the responsive movement of the projector actuator (114), the tracking actuator (116), and the sensor actuator (118). Such delays may produce parallax errors for each viewer. In some embodiments, the local controller (202) may implement a prediction algorithm to predict the movement of the user (102) and adjust the various actuators accordingly to reduce parallax. In some embodiments, the local controller (202) and the remote controller (252) implement a low bandwidth and low latency communication channel for transmitting position information separately from the video stream.

[0052] The algorithm for latency compensation may include a technique for estimating the error between the perspective of a received video stream and an ideal perspective based on tracking data from a local tracking system (106). Because the local user (102) may move, the perspective at which the received video stream is captured may be different from the actual perspective of the local user when the image is provided to the local user. To compensate for this possibility, an error estimation technique may be used to correct the difference between the actual perspective of the received video stream (which is based on historical data from the local tracking system (106)) and the actual current perspective of the local user. For example, a local controller (202) receives a video stream from a remote controller (252) and receives tracking information of the local user (102) from the tracking system (106). The local controller (202) estimates the error between the two perspectives and uses the error to shift (e.g., distort or otherwise adjust) the received image to a correct average position. The image may be shifted by tilting or adjusting the projector (104) using a projector actuator (114). In some embodiments, the image is shifted by shifting pixels within a video image projected by the projector (104). If the local user (102) stops moving, the position error is zero and the projected image is re-centered with true parallax. In some embodiments, the image sensor device (108) captures an area wider than the projection to account for shifts in the projected image, thereby avoiding border clipping.

[0053] In some cases, the video conferencing booth (100) can be used to view three-dimensional stereoscopic video without the presence of a remote user. For example, the projector (104) projects a video having a left eye image and a right eye image. The tracking system (106) continues to monitor the movement of the user (102) while viewing the three-dimensional stereoscopic video. The local controller (202) controls the position of the projector (104) based on the movement of the user (102) by actuating the projector actuator (114). In such an embodiment, the image sensor device (108) (and the corresponding sensor actuator (118)) can be disabled or omitted because data may not be transmitted to the remote controller (252). In addition, in such an embodiment, one direction of the video pipeline (e.g., one communication direction between the local controller (202) and the remote controller (252)) can be disabled or omitted.

[0054] In some embodiments, a 3D autostereoscopic monitor may be implemented instead of using a stereoscopic projector and a retroreflective screen. The position of the image sensor device (108) is still controlled by the sensor actuator (118) to track the position of the local user (102). The output of the 3D autostereoscopic monitor may be adjusted based on the position of the local user (102) provided by the image sensor device (108). In such an embodiment, the projector (104), the mirror (120), and the splitter (122) may be omitted.

[0055] In some cases, the output of the projector (104) can be gated so that the output of the projector (104) does not interfere with the operation of the image sensor device (108). As an example, the image sensor device (108) can only sense incident light when the projector (104) is not providing output. In some embodiments, the image sensor device (108) can include a shutter that is closed when the projector (104) is emitting light (i.e., in the on phase of the duty cycle), but can be open when the projector (104) is not emitting light (i.e., in the off phase of the duty cycle). In some embodiments, pixels within the image sensor device (108) can be reset after the projector (104) emits a first light, and then light is collected in a time period before the projector (104) emits a second light. With this type of arrangement, the duty cycle or frequency of the gate can be selected so that the output of the projector (104) is mostly or completely out of alignment with the operation of the image sensor device (108).

[0056] Other video conferencing configurations

[0057] Figure 6Another example video conferencing booth (600) is illustrated. The video conferencing booth (600) includes a local user (602) and a plurality of image sensor devices (608) (i.e., n image sensor devices), each image sensor device (608) being associated with a corresponding remote user. An n-th image sensor device of the n-th image sensor devices includes a first image sensor of the n-th image sensor device and a second image sensor of the n-th image sensor device. The first image sensor of the n-th image sensor device is configured to generate a video stream corresponding to a left eye of the n-th remote user, and the second image sensor of the n-th image sensor device is configured to generate a video stream corresponding to a right eye of the n-th remote user. Figure 6 The example includes a first image sensor device (608A), a second image sensor device (608B), and a third image sensor device (608C) associated with a first remote user, a second remote user, and a third remote user, respectively. Each remote user is located in a corresponding remote video conference booth. Although Figure 6 Three remote users are provided, but for a desired number of remote participants, the number of multiple image sensor devices (608) can be reduced (e.g., reduced to two image sensor devices (608)) or increased (e.g., increased to five image sensor devices (608)). Additionally, in other cases, the position of the local user (602) may be different. For example, when five remote users are involved, the first image sensor device may be located to the left of the local user (602), the second image sensor device may be located diagonally to the left of the local user (602), the third image sensor device may be located in front of the local user (602), the fourth image sensor device may be located to the diagonally right of the local user (602), and the fifth image sensor device may be located to the right of the local user (602).

[0058] A local user (602) and a plurality of image sensor devices (608) may be positioned around a table (610). The local user (602) is positioned in front of a retroreflective screen (624). The local user (602) may wear a wearable stereo projector (604) configured to project images onto the screen (624). In particular, as an example, the wearable stereo projector (604) projects a left eye image and a right eye image onto the screen (624). The screen (624) reflects the left eye image onto the left eye of the local user (602) and reflects the right eye image onto the right eye of the local user (602). Since the retroreflective screen reflects light in the direction in which the light is received, the projection from the wearable stereo projector (604) is directed into the eyes of the local user (602) regardless of the gaze direction of the local user (602). The screen (624) is configured such that the local user (602) cannot see the plurality of image sensor devices (608). Additionally, the screen (624) can be configured as a one-way screen, enabling multiple image sensor devices (608) to capture images and video of the local user (602). For example, in some cases, the screen (624) includes perforations, allowing multiple image sensor devices (608) to view through the screen (624).

[0059] In some examples, the output of the wearable stereo projector (604) can be gated so that the output of the projector (604) does not interfere with the operation of the plurality of image sensor devices (608). As an example, when the wearable stereo projector (604) does not provide output, the plurality of image sensor devices (608) can only sense incident light. In some embodiments, the plurality of image sensor devices (608) can include a shutter that is closed when the wearable stereo projector (604) emits light (i.e., is in the open phase of the duty cycle), but can be open when the wearable stereo projector (604) does not emit light (i.e., is in the closed phase of the duty cycle). In some embodiments, pixels within the plurality of image sensor devices (608) can be reset after the wearable stereo projector (604) emits a first light, and then light is collected during a period of time before the wearable stereo projector (604) emits a second light. With this type of arrangement, the duty cycle or frequency of the gating can be selected so that the output of the wearable stereoscopic projector (604) is mostly or completely out of alignment with the operation of the multiple image sensor devices (608).

[0060] In some other examples, the screen (624) can be transparent to the first polarization and reflective to the second polarization. In such examples, the wearable stereoscopic projector (604) can include a polarization filter so that the wearable stereoscopic projector (604) only emits light of the second polarization. If desired, one or more of the image sensor devices (608) can include a polarization filter to block light of the second polarization.

[0061] In addition, the video conference booth (600) includes a tracking system (not shown) that tracks the head and facial movements of the local user (602). In some cases, the tracking system is an infrared or near-infrared tracking system. In this case, a light source can be implemented and projected onto the local user (602) to assist in depth determination. In one example, the infrared tracking system uses a marker (such as a reflective marker coupled to the local user (602)) to track the movement of the local user (602). In another example, the tracking system is configured to track the movement of the local user (602) using an eye tracking algorithm, a facial tracking algorithm, and other algorithms that may be machine learning algorithms. In a further example, the tracking system is embedded on or within a wearable stereo projector (604). For example, one or more motion sensors (and / or orientation sensors) may be located within the wearable stereo projector (604) to track the head position of the local user (602). When the head orientation of the local user (602) changes, the motion and / or orientation sensors detect the movement and / or orientation of the head of the local user (602). The wearable stereo projector (604) transmits a signal indicating orientation to each connected remote video conferencing booth. If desired, the tracking system can use images from one or more of the image sensor devices (608), optionally together with information about the positions of a plurality of sensor actuators (618) (described below), to track the head and facial movements of the local user (602).

[0062] Each of the plurality of image sensor devices (608) is located on a sensor actuator (618). For example, a first image sensor device (608A) is located on a first sensor actuator (618A), a second image sensor device (608B) is located on a second sensor actuator (618B), and a third image sensor device (608C) is located on a third sensor actuator (618C). Each image sensor device (608) is associated with a remote user in a remote video conference booth. For example, as previously described with respect to Figure 2 As described, a plurality of remote compartment controllers (252) can be connected to the local compartment controller (202) via the network (230), with at least one remote compartment controller (252) for each remote user. The movement of each remote user is tracked by a corresponding tracking system. The local compartment controller (e.g. Figure 2The local booth controller (202) in the embodiment receives signals indicative of the movement of each remote user and controls the sensor actuator (618) associated with the respective remote user to actuate the respective image sensor device (608). In this manner, each image sensor device (608) moves in conjunction with the movement of the associated remote user, thereby mimicking their movement and matching the orientation of the image sensor device (608) with the orientation of the respective remote user.

[0063] Figure 7 A method (700) for controlling components of a video conferencing booth (600) is provided. The method (700) may be performed, for example, by a local booth controller (202) or a remote booth controller (252). In addition, Figure 7 The steps provided in are merely examples and may be performed in a different order or simultaneously.

[0064] At step (702), the local controller (202) obtains a video stream from a remote booth controller (252). For example, each image sensor device (608) is associated with a specific user. The image sensor device (608) transmits the captured video to the corresponding user. The first image sensor device (608A) transmits the video of the local user (602) to the first remote booth controller, the second image sensor device (608B) transmits the video of the local user (602) to the second remote booth controller, and the third image sensor device (608C) transmits the video of the local user (602) to the third remote booth controller. The local controller (202) receives the video stream from the associated image sensor device (608) in each remote video conference booth to obtain the video of each remote user.

[0065] At step (704), the local controller (202) projects the video streams onto the screen (624) using the wearable stereo projector (604). For example, the video streams showing each remote user may be combined into a single video stream (e.g., a 180 degree video) and projected onto the screen (624). When the local user (602) turns his head or adjusts his gaze, the projected video is adjusted to show the corresponding remote user based on the detected viewing angle of the user (102). For example, when the local user (602) faces the first remote user (e.g., Figure 6 When the local user (602) faces the second remote user (e.g., Figure 6When the local user (602) faces a third remote user (e.g., Figure 6 When the wearable stereo projector (604) projects a video corresponding to 0 to 90 degrees of the 180-degree video, the wearable stereo projector (604) projects a video corresponding to 0 to 90 degrees of the 180-degree video. In this case, the corresponding image sensor device associated with the local user (602) may have a large enough field of view (FOV) to capture all remote participants (e.g., approximately 180 degrees or greater).

[0066] In another embodiment, the wearable stereo projector (604) projects a single video stream based on the viewing direction of the local user (602). For example, the local controller (202) can "stitch" together each video received from the remote image sensor device to create a single continuous image (e.g., combining three 60-degree videos into a single 180-degree video). When the local user (602) faces the first remote user, the wearable stereo projector (604) projects the video received from the corresponding remote image sensor device that captured the first remote user. When the local user (602) faces the second remote user, the wearable stereo projector (604) projects the video received from the corresponding remote image sensor device that captured the second remote user. When the local user (602) faces the third remote user, the wearable stereo projector (604) projects the video received from the corresponding remote image sensor device that captured the third remote user. Only a single remote user can be shown at a given time. In such a case, the corresponding image sensor device associated with the local user (602) may have a FOV (e.g., approximately 60 degrees) that can only capture a single remote user at a time. Additionally, in such an embodiment, lens distortion may be biased toward the center area of ​​the projected image while transmitting lower angular resolution images in the periphery (e.g., above 40 degrees from the center of the projected image). To account for differences within each video from the remote image sensor device, the local controller (202) may remove, replace, or otherwise alter the background of each video.

[0067] At step (706), the local controller (202) receives a tracking stream for each remote user from each remote booth controller (252). For example, a tracking system associated with each remote user transmits a signal to the local controller (202) indicating the movement of the corresponding remote user. In some cases, the local controller (202) also transmits a signal indicating the local user (602) to each remote booth controller (252).

[0068] At step (708), the local controller (202) controls the plurality of sensor actuators (618) to actuate the plurality of image sensor devices (608) based on the received tracking streams, for example, controlling the first sensor actuator (618A) based on the movement of the first remote user, controlling the second sensor actuator (618B) based on the movement of the second remote user, and controlling the third sensor actuator (618C) based on the movement of the third remote user.

[0069] Similar to Figure 1 In the example video conferencing booth (100) of FIG. 1 , the video conferencing booth (600) may also experience delays due to network latency. Such delays may produce parallax errors for each viewer. In some embodiments, the local controller (202) implements a prediction algorithm to predict the movement of the user (102) and ensure that the wearable stereo projector (604) projects the correct video stream. For example, the local controller (202) may predict that the local user (602) is turning from facing the first image sensor device (608A) to the second image sensor device (608B). In some embodiments, the local controller (202) may distort the projected video to account for this movement, thereby adjusting the viewing angle without waiting for new position information to be transmitted to the corresponding remote sensor actuator (618) before the wearable stereo projector (604) receives and projects a new video frame.

[0070] Figure 1 and Figure 6 The embodiments of the present invention provide only a single local user. However, the aspects described herein can be extended to a video conference table for multiple local users. For example, FIG. 8A to FIG. 8B An example video conference room (800) is illustrated with multiple local users (802) at a table (801). Each local user (802) has an associated wearable stereo projector (804). FIG. 8A to FIG. 8B In the example of , the plurality of local users (802) include a first local user (802A) having a first wearable stereo projector (804A), a second local user (802B) having a second wearable stereo projector (804B), a third local user (802C) having a third wearable stereo projector (804C), and a fourth local user (802D) having a fourth wearable stereo projector (804D). However, in some cases, fewer or more local users (802) may be located at the table (801).

[0071] A plurality of image sensor devices (808) are located opposite a table (801) of a plurality of local users (802) and behind a screen (824). Each image sensor device (808) is located on a corresponding sensor actuator (818). Similar to Figure 1 and Figure 6 , each image sensor device (808) is associated with a remote user participating in the video conference. Thus, in the example of FIG8 , eight people are participating in the video conference: a plurality of local users (802), and a plurality of remote users (808) represented by a plurality of image sensor devices. A remote video conference table (not shown) includes a remote image sensor device associated with each local user (802).

[0072] Each image sensor device (808) captures a view associated with a remote user as a remote video conference table. For example, the first image sensor device (808A) captures the view of the first remote user, the second image sensor device (808B) captures the view of the second remote user, the third image sensor device (808C) captures the view of the third remote user, and the fourth image sensor device (808D) captures the view of the fourth remote user. Each remote user wears a remote wearable stereo projector (not shown). The movement of the remote user's head can be tracked (e.g., using a wearable projector, using an image sensor device, using another head tracking system, etc.). The sensor actuator (818) is controlled to move each image sensor device (808) and imitate or reflect the movement of its corresponding remote user. In FIG. 8A to FIG. 8B In the example of , the second local user (802B) is the main speaker. The second image sensor device (808B), the third image sensor device (808C), and the fourth image sensor device (808D) are all focused on (e.g., facing) the second local user (802B). The first image sensor device (808A) has an independent focus and is focused on the first local user (802A).

[0073] In some embodiments, the multiple image sensor devices (808) are enclosed within the blind 810, hiding the multiple image sensor devices (808) from the multiple local users (802). In addition, the screen (824) can be configured as a one-way screen, enabling the multiple image sensor devices (808) to capture images and videos of the multiple local users (802). For example, in some cases, the screen (824) includes perforations, thereby allowing the multiple image sensor devices (808) to view through the screen (824). As another example, the screen (824) can be reflective to a first polarization and transparent to a second polarization, while the wearable stereo projector (804) is equipped with a first polarization filter to emit only light of the first polarization, and the multiple image sensor devices (808) are optionally equipped with a second polarization filter to receive only light of the second polarization.

[0074] In various examples, the outputs of the wearable stereo projector (802) can be gated together so that their outputs do not interfere with the operation of the multiple image sensor devices (808). The outputs of the wearable stereo projector (804) can be gated so that the outputs of the wearable stereo projector (804) do not interfere with the operation of the multiple image sensor devices (808). For example, when the wearable stereo projector (804) does not provide output, the multiple image sensor devices (808) can only sense incident light. In some embodiments, the multiple image sensor devices (808) can include a shutter that is closed when the wearable stereo projector (804) emits light (i.e., is in the off phase of the duty cycle), but when the wearable stereo projector (804) does not emit light (i.e., is in the off phase of the duty cycle), the shutter can be open. In some embodiments, the pixels within the multiple image sensor devices (808) can be reset after the wearable stereo projector (804) emits a first light, and then collect light in a time period before the wearable stereo projector (804) emits a second light. With this type of arrangement, the duty cycle or frequency of the gating can be selected so that the output of the wearable stereoscopic projector (804) is mostly or completely out of alignment with the operation of the multiple image sensor devices (808).

[0075] Similarly, the remote image sensor device captures a view (802) of the remote video conference table for each local user. Each wearable stereo projector (804) projects a view of the remote image sensor device associated with the corresponding local user (802). Due to the retroreflective nature of the screen (824), the projection from the wearable stereo projector (804) is only seen by the corresponding local user (802). For example, the projection from the first wearable stereo projector (804A) can only be viewed by the first local user (802A), the projection from the second wearable stereo projector (804B) can only be viewed by the second local user (804B), and so on. In some cases, when the local user (802) turns his gaze to another local user (802), the wearable stereo projector (804) dims or turns off its projection. For example, when the second local user (802B) turns toward the first local user (802A) or the third local user (802C), the second wearable stereoscopic projector (804B) adjusts the projection to dim or turn off.

[0076] In some cases, the video conference booth (100), video conference booth (600), and / or video conference table (800) may include an "observer" camera that is shared by inactive viewers of an ongoing video conference. Such an observer camera may be located at a higher vantage point than the active participants and may not be movable. In some cases, who is an "active" participant and who is an "inactive" participant may change dynamically based on participation in the video conference (e.g., who is talking or changing gaze direction).

[0077] Additional features of the video conferencing booth (600) and the video conferencing table (800) may be understood based on the configuration of the video conferencing booth (100). For example, the video conferencing booth (600) and the video conferencing table (800) may include microphones and speakers (e.g., spatial audio equipment) for transmitting audio data. Embodiments of the video conferencing booth (600) and the video conferencing table (800) are not limited to the respective Figure 6 and FIG. 8A to FIG. 8B The embodiment shown in FIG.

[0078] The above-mentioned system and method can provide a video conference booth.The system, method and device according to the present disclosure can adopt any one or more of the following configurations.

[0079] (1) A video conferencing booth, comprising: a tracking system configured to track the position of a first user; a stereo projector; a first actuator configured to translate the stereo projector along at least a first axis; an image sensor device including a first image sensor configured to generate a video stream corresponding to a left eye of a second user of a remote video conferencing booth, and a second image sensor configured to generate a video stream corresponding to a right eye of the second user of the remote video conferencing booth; a second actuator configured to translate the image sensor device along at least a second axis; and a controller configured to: obtain a first stream of a first position of the first user from the tracking system, transmit a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first position of the first user, transmit the first stream of the first position of the first user to the remote video conferencing booth, receive a second stream of a second position of the second user of the remote video conferencing booth from the remote video conferencing booth, and transmit a command to the second actuator to adjust the position of the image sensor device along the second axis based on the second stream of the second position of the second user.

[0080] (2) The video conferencing booth according to (1), further comprising a retroreflective screen, wherein the stereo projector is configured to project a left eye image and a right eye image, wherein the controller is configured to transmit a command to adjust the position of the stereo projector along at least the first axis so that the left eye image from the projector is reflected by the retroreflective screen toward the left eye of the first user, and the right eye image from the projector is reflected by the retroreflective screen toward the right eye of the first user.

[0081] (3) The video conferencing booth according to (2), further comprising a polarization beam splitter configured to reflect light of a first polarization and transmit light of a second polarization, wherein the polarization beam splitter is positioned (A) such that the light of the first polarization from the first user is reflected from a first surface of the polarization beam splitter and directed to the image sensor device, (B) such that the light of the first polarization from the stereo projector is reflected from a second surface of the polarization beam splitter and directed to the retroreflective screen, and (C) such that the light of the second polarization from the retroreflective screen is transmitted through the polarization beam splitter and directed to the first user.

[0082] (4) The video conferencing booth of (3), wherein the retroreflective screen includes a quarter wave plate so that the first polarized light reflected from the retroreflective screen is converted into the second polarized light.

[0083] (5) The video conferencing booth according to any one of (3) to (4), further comprising at least one polarization filter configured to pass the first polarized light and block the second polarized light, wherein the at least one polarization filter comprises (1) a first polarization filter disposed in the optical path between the polarization beam splitter and the image sensor device, and / or (2) a second polarization filter disposed in the optical path between the polarization beam splitter and the stereo projector.

[0084] (6) The video conferencing booth of any one of (1) to (5), wherein the tracking system includes a head tracking system configured to track a head position of the first user.

[0085] (7) The video conferencing booth of any one of (1) to (6), wherein the tracking system comprises a facial tracking system configured to track a facial position of the first user.

[0086] (8) The video conferencing booth of any one of (1) to (7), wherein at least the first axis includes three axes, and wherein at least the second axis includes three axes.

[0087] (9) The video conferencing booth according to any one of (1) to (8), further comprising a spatial audio device configured to project sound provided by the remote video conferencing booth.

[0088] (10) A video conferencing booth according to any one of (1) to (9), wherein the stereo projector has a duty cycle with an on phase and an off phase, wherein the stereo projector is configured to project light during the on phase of the duty cycle of the stereo projector but not the off phase, and wherein the image sensor device is configured to capture an image of the first user during the off phase of the duty cycle of the stereo projector.

[0089] (11) A video conferencing booth according to any one of (1) to (10), wherein the controller is further configured to: receive a video stream of the second user from the remote video conferencing booth, wherein the video stream of the second user includes at least a given frame captured based on the first user being at a third position, determine that the first user is at a fourth position when receiving the given frame of the video stream at least in part based on the first stream of the first user's first position from the tracking system, estimate an error between the third position of the first user and the fourth position of the first user, and adjust the video stream of the second user based on the estimated error.

[0090] (12) A video conferencing booth according to any one of (1) to (11), wherein the controller is further configured to: obtain a third stream of a third user's third position from the tracking system, transmit a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first user's first position and the third stream of the third user's third position, and transmit the first stream of the first user's first position and the third stream of the third user's third position to the remote video conferencing booth.

[0091] (13) The video conferencing booth according to any one of (1) to (12), further comprising: a second image sensor; and a third actuator configured to translate the second image sensor along at least a third axis; wherein the controller is further configured to: transmit the first stream of the first position of the first user to a second remote video conferencing booth, receive a third stream of the third position of the third user of the second remote video conferencing booth from the second remote video conferencing booth, and transmit a command to the third actuator to adjust the position of the second image sensor along the third axis based on the third stream of the third position of the third user.

[0092] (14) A method for operating a video conferencing booth, the video conferencing booth comprising: a tracking system configured to track the position of a first user, a stereo projector; a first actuator configured to translate the stereo projector along at least a first axis; an image sensor device comprising a first image sensor configured to generate a video stream corresponding to a left eye of a second user of the remote video conferencing booth, and a second image sensor configured to generate a video stream corresponding to a right eye of the second user of the remote video conferencing booth; and a second actuator configured to translate the image sensor device along at least a second axis, the method comprising: obtaining a first stream of a first position of the first user from the tracking system, transmitting a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first position of the first user, transmitting the first stream of the first position of the first user to the remote video conferencing booth, receiving a second stream of a second position of the second user of the remote video conferencing booth from the remote video conferencing booth, and transmitting a command to the second actuator to adjust the position of the image sensor device along the second axis based on the second stream of the second position of the second user.

[0093] (15) A method according to (14), wherein the video conferencing booth includes a retroreflective screen, and wherein the method further includes: projecting a left eye image and a right eye image using the stereo projector, and transmitting a command to adjust the position of the stereo projector along at least a first axis so that the left eye image from the projector is reflected by the retroreflective screen toward the left eye of the first user, and the right eye image from the projector is reflected by the retroreflective screen toward the right eye of the first user.

[0094] (16) The method according to (15) further includes: using a polarization beam splitter to reflect light of a first polarization, and using the polarization beam splitter to transmit light of a second polarization, wherein the polarization beam splitter is positioned to (A) allow the first polarized light from the first user to be reflected from the first surface of the polarization beam splitter and directed to the image sensor device, (B) allow the first polarized light from the stereo projector to be reflected from the second surface of the polarization beam splitter and directed to the retroreflective screen, and (C) allow the second polarized light from the retroreflective screen to pass through the polarization beam splitter and be directed to the first user.

[0095] (17) The method according to (16), wherein the retroreflective screen includes a quarter wave plate so that the first polarized light reflected from the retroreflective screen is converted into the second polarized light.

[0096] (18) A method according to any one of (16) to (17), wherein the video conferencing booth includes at least one polarization filter, which is configured to pass light of the first polarization and block light of the second polarization, wherein the at least one polarization filter includes (1) a first polarization filter disposed in an optical path between the polarization beam splitter and the image sensor device, and / or (2) a second polarization filter disposed in an optical path between the polarization beam splitter and the stereo projector.

[0097] (19) The method according to any one of (14) to (18), further comprising: tracking the head position of the first user using the tracking system.

[0098] (20) The method according to any one of (14) to (19), further comprising: tracking the facial position of the first user using the tracking system.

[0099] (21) The method according to any one of (14) to (20), further comprising: using a spatial audio device to project the sound provided by the remote video conferencing booth.

[0100] (22) A method according to any one of (14) to (21), wherein the stereoscopic projector has a duty cycle with an on phase and an off phase, and the method further comprises: utilizing the stereoscopic projector to project light during the on phase of the duty cycle of the stereoscopic projector rather than the off phase; and utilizing the image sensor device to capture an image of the first user during the off phase of the duty cycle of the stereoscopic projector.

[0101] (23) The method according to any one of (14) to (22), further comprising: receiving a video stream of the second user from the remote video conferencing booth, wherein the video stream of the second user includes at least a given frame captured based on the first user being at a third position, determining that the first user is at a fourth position when receiving the given frame of the video stream at least in part based on the first stream of the first user's first position from the tracking system, estimating an error between the third position of the first user and the fourth position of the first user, and adjusting the video stream of the second user based on the estimated error.

[0102] (24) The method according to any one of (14) to (23), further includes: obtaining a third stream of a third user's third position from the tracking system, transmitting a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first user's first position and the third stream of the third user's third position, and transmitting the first stream of the first user's first position and the third stream of the third user's third position to a remote video conferencing booth.

[0103] (25) A method according to any one of (14) to (24), wherein the video conferencing booth includes a second image sensor device and a third actuator, the third actuator being configured to translate the second image sensor device along at least a third axis, and wherein the method further includes: transmitting the first stream of the first position of the first user to a second remote video conferencing booth, receiving a third stream of the third position of a third user of the second remote video conferencing booth from the second remote video conferencing booth, and transmitting a command to the third actuator to adjust the position of the second image sensor device along the third axis based on the third stream of the third position of the third user.

[0104] (26) A non-transitory computer-readable medium storing instructions that, when executed by a processor of a projection system, cause the projection system to perform operations including the method according to any one of (14) to (25).

[0105] (27) A video conference booth comprising: a stereo projector configured to project a first image and a second image; a polarizing beam splitter configured to reflect light of a first polarization and transmit light of a second polarization; a retroreflective screen; and a camera configured to monitor a user located within the video conference booth, wherein the polarizing beam splitter is positioned (A) such that light of the first polarization from the user located within the video conference booth is reflected from a first surface of the polarizing beam splitter and directed toward the camera, (B) such that light of the first polarization from the stereo projector is reflected from a second surface of the polarizing beam splitter and directed toward the retroreflective screen, and (C) such that light of the second polarization from the retroreflective screen is transmitted through the polarizing beam splitter and directed toward the user.

[0106] (28) A video conference table comprising: a tracking system configured to track the positions of a first local user and a second local user; a first image sensor device configured to capture at least the first local user; a second image sensor device configured to capture at least the second local user; a first actuator configured to translate the first image sensor device along at least a first axis; a second actuator configured to translate the second image sensor device along at least a second axis; and a controller configured to: receive a first stream of a first position of a first remote user, transmit a command to the first actuator to adjust the position of the first image sensor device along the first axis based on the first stream of the first position of the first remote user, receive a second stream of a second position of a second remote user, and transmit a command to the second actuator to adjust the position of the second image sensor device along the second axis based on the second stream of the second position of the second remote user.

[0107] (29) The video conference table according to (28) further includes: a reflective screen; and a first wearable stereo projector, which is configured to project a left eye image and a right eye image, wherein the first wearable stereo projector projects the left eye image and the right eye image so that the left eye image is reflected by the reflective screen toward the left eye of the first local user, and the right eye image is reflected by the reflective screen toward the right eye of the first local user.

[0108] (30) The video conferencing table of (29), wherein the tracking system comprises a motion sensor embedded in the first wearable stereo projector, the motion sensor being configured to track the head position of the first local user.

[0109] (31) A video conference table according to (30), wherein the controller is configured to: detect movement of the first local user's head toward the second local user based on a signal from the motion sensor, and dim the left eye image and the right eye image projected by the first wearable stereo projector in response to the movement.

[0110] (32) A video conferencing table according to any one of (29) to (31), wherein the first wearable stereoscopic projector includes an eyeglass frame, a left eye projector located on a first side of the eyeglass frame, and a right eye projector located on a second side of the eyeglass frame.

[0111] (33) A video conferencing table according to any one of (29) to (32), wherein the first wearable stereo projector has a duty cycle with an on phase and an off phase, wherein the first wearable stereo projector is configured to project light during the on phase of the duty cycle of the first wearable stereo projector but not the off phase, and wherein the first image sensor device is configured to capture an image of the first local user during the off phase of the duty cycle of the first wearable stereo projector.

[0112] (34) The video conference table according to (33) further includes a second wearable stereo projector, the second wearable stereo projector having a duty cycle with an on phase and an off phase, wherein the second wearable stereo projector is configured to project light during the on phase of the duty cycle of the second wearable stereo projector rather than the off phase, and wherein the second image sensor device is configured to capture an image of the second local user during the off phase of the duty cycle of the second wearable stereo projector.

[0113] (35) A video conferencing table according to any one of (29) to (34), wherein the retroreflective screen is configured to reflect light of a first polarization and pass light of a second polarization, and wherein the first wearable stereo projector includes a first polarization filter to emit only light of the first polarization, and wherein the first image sensor device and the second image sensor device each include a second polarization filter to receive only light of the second polarization.

[0114] (36) A video conferencing table according to any one of (28) to (35), wherein at least the first axis includes three axes, and wherein at least the second axis includes three axes.

[0115] (37) A method for operating a video conference table, the video conference table comprising: a tracking system configured to track the positions of a first local user and a second local user; a first image sensor device configured to capture at least the first local user; a second image sensor device configured to capture at least the second local user; a first actuator configured to translate the first image sensor device along at least a first axis; and a second actuator configured to translate the second image sensor device along at least a second axis, the method comprising: receiving a first stream of a first position of a first remote user, transmitting a command to the first actuator to adjust the position of the first image sensor device along the first axis based on the first stream of the first position of the first remote user, receiving a second stream of a second position of a second remote user, and transmitting a command to the second actuator to adjust the position of the second image sensor device along the second axis based on the second stream of the second position of the second remote user.

[0116] (38) A method according to (37), wherein the video conference table further includes a reflective screen and a first wearable stereo projector, and wherein the method further includes: using the first wearable stereo projector to project a left eye image so that the left eye image is reflected by the reflective screen toward the left eye of the first local user, and using the first wearable stereo projector to project a right eye image so that the right eye image is reflected by the reflective screen toward the right eye of the first local user.

[0117] (39) The method according to (38) further includes: tracking the position of the first local user using a motion sensor embedded in the first wearable stereo projector.

[0118] (40) The method according to (39) further includes: detecting the movement of the head of the first local user toward the second local user based on the signal from the motion sensor, and responding to the movement and dimming the left eye image and the right eye image projected by the first wearable stereo projector using the first wearable stereo projector.

[0119] (41) A method according to any one of (38) to (40), wherein the wearable stereoscopic projector includes an eyeglass frame, a left-eye projector located on a first side of the eyeglass frame, and a right-eye projector located on a second side of the eyeglass frame.

[0120] (42) A method according to any one of (38) to (41), wherein the first wearable stereo projector has a duty cycle with an on phase and an off phase, and the method further comprises: using the first wearable stereo projector to project light during the on phase of the duty cycle of the first wearable stereo projector rather than the off phase, and using the first image sensor device to capture an image of the first local user during the off phase of the duty cycle of the second wearable stereo projector.

[0121] (43) A method according to (42), wherein the video conference table further includes a second wearable stereo projector, the second wearable stereo projector having a duty cycle with an on phase and an off phase, and the method further includes: using the second wearable stereo projector to project light during the on phase of the duty cycle of the second wearable stereo projector instead of the off phase, and using the second image sensor device to capture the image of the second local user during the off phase of the duty cycle of the second wearable stereo projector.

[0122] (44) The method according to any one of (38) to (43) further includes: using the retroreflective screen to only reflect light of the first polarization, using the retroreflective screen to only pass light of the second polarization, using the first wearable stereo projector to only emit light of the first polarization, and using the first image sensor device and the second image sensor device to only receive light of the second polarization.

[0123] (45) The method according to any one of (37) to (44), wherein at least the first axis includes three axes, and wherein at least the second axis includes three axes.

[0124] (46) A video conferencing booth, comprising: a tracking system configured to track the position of a first user; a wearable stereo projector; an image sensor device; an actuator configured to translate the image sensor device along at least a first axis; and a controller configured to: obtain a first stream of a first position of the first user from the tracking system, transmit the first stream of the first position of the first user to a remote video conferencing booth, receive a second stream of a second position of a second user of the remote video conferencing booth from the remote video conferencing booth, and transmit a command to the actuator to adjust the position of the image sensor device along the first axis based on the second stream of the second position of the second user.

[0125] (47) The video conferencing booth according to (46) further includes a retroreflective screen, wherein the wearable stereo projector is configured to project a left eye image and a right eye image, so that the left eye image from the wearable stereo projector is reflected by the retroreflective screen toward the left eye of the first user, and the right eye image from the wearable stereo projector is reflected by the retroreflective screen toward the right eye of the first user.

[0126] (48) A video conferencing booth according to any one of (46) to (47), wherein the tracking system is embedded in the wearable stereo projector.

[0127] (49) A video conferencing booth according to any one of (46) to (48), wherein the tracking system includes a head tracking system configured to track the head position of the first user.

[0128] (50) A video conferencing booth according to any one of (46) to (49), wherein the controller is further configured to: receive a video stream of the second user from the remote video conferencing booth, wherein the video stream of the second user includes at least a given frame captured based on the first user being at a third position, determine that the first user is at a fourth position when receiving the given frame of the video stream at least in part based on the first stream of the first user's first position from the tracking system, estimate an error between the third position of the first user and the fourth position of the first user, and adjust the video stream of the second user based on the estimated error.

[0129] (51) A video conferencing booth according to any one of (46) to (50), wherein at least the first axis includes three axes, and wherein at least the second axis includes three axes.

[0130] (52) A video conferencing booth according to any one of (46) to (51), wherein the controller is further configured to: obtain a third stream of a third user's third position from the tracking system, and transmit the first stream of the first user's first position and the third stream of the third user's third position to the remote video conferencing booth.

[0131] (53) A video conferencing booth according to any one of (46) to (52), wherein the wearable stereo projector has a duty cycle with an on phase and an off phase, wherein the wearable stereo projector is configured to project light during the on phase of the duty cycle of the wearable stereo projector but not the off phase, and wherein the image sensor device is configured to capture an image of the first user during the off phase of the duty cycle of the wearable stereo projector.

[0132] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of these processes, etc. have been described as being performed in a particular ordered order, these processes can be practiced with the described steps performed in an order different from that described herein. It should further be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0133] Therefore, it should be understood that the above description is intended to be illustrative and not restrictive. By reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but should be determined with reference to the full range of equivalents to which the appended claims and these claims are eligible to be considered. It is expected and hoped that the technology discussed herein will develop in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In a word, it should be understood that the present application is capable of modification and change.

[0134] All terms used in the claims are intended to be given their broadest reasonable interpretations and ordinary meanings as understood by those skilled in the art described herein, unless an explicit indication to the contrary appears herein. In particular, the use of singular articles such as "a", "the", "said", etc. should be understood to recite one or more of the indicated elements, unless a claim recites an explicit limitation to the contrary. As used herein, "and / or" means that one, more than one, or all of the situations may occur.

[0135] The abstract of the present disclosure is provided to allow the reader to quickly determine the nature of the technical disclosure. This abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description, it can be seen that various features are combined together into various embodiments for the purpose of connecting the present disclosure as a whole. The method of the present disclosure should not be interpreted as reflecting the intention that the claimed embodiments incorporate more features than those explicitly recited in each claim. On the contrary, as reflected in the attached claims, the inventive subject matter lies in less than all the features of a single disclosed embodiment. Therefore, the attached claims are hereby incorporated into the detailed description, and each claim stands alone as a separately claimed subject matter.

Claims

1. A video conference booth, comprising: a tracking system configured to track a location of the first user; Stereoscopic projector; a first actuator configured to translate the stereoprojector along at least a first axis; an image sensor device comprising a first image sensor configured to generate a video stream corresponding to a left eye of a second user of a remote video conferencing booth, and a second image sensor configured to generate a video stream corresponding to a right eye of the second user of the remote video conferencing booth; a second actuator configured to translate the image sensor device along at least a second axis; as well as A controller configured to: obtaining from the tracking system a first stream of a first location of the first user, transmitting a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first position of the first user, transmitting the first stream of the first user's first location to the remote video conferencing booth, receiving from the remote video conferencing booth a second stream of a second location of the second user of the remote video conferencing booth, transmitting a command to the second actuator to adjust the position of the image sensor device along the second axis based on the second stream of the second position of the second user, receiving a video stream of the second user from the remote video conference booth, wherein the video stream of the second user includes at least a given frame captured based on the first user being in a third position, determining, based at least in part on the first stream of the first location of the first user from the tracking system, that the first user is located at a fourth location when the given frame of the video stream is received, estimating an error between the third position of the first user and the fourth position of the first user, and The video stream of the second user is adjusted based on the estimated error.

2. The video conferencing booth of claim 1, further comprising a retroreflective screen, wherein: The stereoscopic projector is configured to project a left-eye image and a right-eye image, wherein the controller is configured to transmit a command to adjust the position of the stereoscopic projector along at least the first axis so that the left-eye image from the projector is reflected by the retroreflective screen toward the left eye of the first user, and the right-eye image from the projector is reflected by the retroreflective screen toward the right eye of the first user.

3. The video conferencing booth of claim 1 or claim 2, further comprising a polarizing beam splitter configured to reflect light of a first polarization and transmit light of a second polarization, wherein The polarization beam splitter is positioned so that (A) the first polarized light from the first user is reflected from the first surface of the polarization beam splitter and directed to the image sensor device, (B) the first polarized light from the stereo projector is reflected from the second surface of the polarization beam splitter and directed to the retroreflective screen, and (C) the second polarized light from the retroreflective screen is transmitted through the polarization beam splitter and directed toward the first user.

4. The video conferencing booth according to any one of claims 1 to 3, wherein: The retroreflective screen includes a quarter wave plate such that light of the first polarization reflected from the retroreflective screen is converted to light of the second polarization.

5. The video conferencing booth of any one of claims 1 to 4, further comprising at least one polarization filter configured to pass light of the first polarization and block light of the second polarization, wherein The at least one polarization filter includes (1) a first polarization filter disposed in an optical path between the polarization beam splitter and the image sensor device, and / or (2) a second polarization filter disposed in an optical path between the polarization beam splitter and the stereo projector.

6. The video conferencing booth according to any one of claims 1 to 5, wherein: The tracking system includes a head tracking system configured to track a head position of the first user.

7. The video conferencing booth according to any one of claims 1 to 6, wherein: Said at least the first axis comprises three axes, and wherein at least the second axis comprises three axes.

8. The video conferencing booth according to any one of claims 1 to 7, wherein: The stereo projector has a duty cycle with an on phase and an off phase, wherein the stereo projector is configured to project light during the on phase of the duty cycle of the stereo projector but not the off phase, and wherein the image sensor device is configured to capture an image of the first user during the off phase of the duty cycle of the stereo projector.

9. The video conferencing booth as claimed in any one of claims 1 to 8, further comprising: a second image sensor device; as well as a third actuator configured to translate the second image sensor device along at least a third axis; Wherein, the controller is further configured as follows: transmitting the first stream of the first user's first location to a second remote video conferencing booth, receiving from the second remote video conferencing booth a third stream of a third location of a third user of the second remote video conferencing booth, and A command is transmitted to the third actuator to adjust the position of the second image sensor device along the third axis based on the third flow of a third position of the third user.

10. A method for operating a video conferencing booth, the video conferencing booth comprising: a tracking system configured to track a location of the first user; Stereoscopic projector; a first actuator configured to translate the stereoprojector along at least a first axis; an image sensor device comprising a first image sensor configured to generate a video stream corresponding to a left eye of a second user of the remote video conferencing booth, and a second image sensor configured to generate a video stream corresponding to a right eye of the second user of the remote video conferencing booth; and a second actuator configured to translate the image sensor device along at least a second axis, the method comprising: obtaining from the tracking system a first stream of a first location of the first user, transmitting a command to the first actuator to adjust the position of the stereo projector along the first axis based on the first stream of the first position of the first user, transmitting the first stream of the first user's first location to the remote video conferencing booth, receiving from the remote video conferencing booth a second stream of a second location of the second user of the remote video conferencing booth, transmitting a command to the second actuator to adjust the position of the image sensor device along the second axis based on the second stream of the second position of the second user, receiving a video stream of the second user from the remote video conference booth, wherein the video stream of the second user includes at least a given frame captured based on the first user being in a third position, determining, based at least in part on the first stream of the first location of the first user from the tracking system, that the first user is located at a fourth location when the given frame of the video stream is received, estimating an error between the third position of the first user and the fourth position of the first user, and The video stream of the second user is adjusted based on the estimated error.

11. The method of claim 10, wherein: The video conferencing booth includes a retroreflective screen, and wherein the method further comprises: projecting a left-eye image and a right-eye image using the stereo projector, and Transmitting commands to adjust the position of the stereoscopic projector along at least the first axis so that the left eye image from the projector is reflected by the retroreflective screen toward the left eye of the first user and the right eye image from the projector is reflected by the retroreflective screen toward the right eye of the first user.

12. The method of claim 10 or claim 11, further comprising: reflecting the light of the first polarization using a polarizing beam splitter, and transmitting light of a second polarization using the polarization beam splitter, Wherein, the polarization beam splitter is positioned so that (A) the first polarized light from the first user is reflected from the first surface of the polarization beam splitter and directed to the image sensor, (B) the first polarized light from the stereo projector is reflected from the second surface of the polarization beam splitter and directed to the retroreflective screen, and (C) the second polarized light from the retroreflective screen is transmitted through the polarization beam splitter and directed to the first user.

13. The method according to any one of claims 10 to 12, wherein: The retroreflective screen includes a quarter wave plate such that light of the first polarization reflected from the retroreflective screen is converted to light of the second polarization.

14. The method according to any one of claims 10 to 13, wherein: The video conferencing booth includes at least one polarization filter configured to pass light of the first polarization and block light of the second polarization, wherein the at least one polarization filter includes (1) a first polarization filter disposed in an optical path between the polarization beam splitter and the image sensor, and / or (2) a second polarization filter disposed in an optical path between the polarization beam splitter and the stereo projector.

15. A non-transitory computer readable medium storing instructions that, when executed by a processor of a projection system, cause the projection system to perform operations including the method of any one of claims 11 to 14.

Images