An autonomous videoconferencing system with virtual director assistance.

Abstract

A system and method are provided for enhancing video conferencing and remote collaboration using sub-symbolic and symbolic artificial intelligence. The autonomous video conferencing system of the present disclosure comprises one main smart camera and multiple peripheral smart cameras, optionally coupled to one or more smart sensors. Each smart camera is equipped with a vision pipeline supported by machine learning to detect objects and their interactions and associated changes in gestures and postures, and a virtual director adapted to apply a predefined set of rules consistent with TV studio production principles. The main camera is adapted to select and update an attention video stream in real time under the direction of the virtual director, and stream the updated attention stream to a user computer. A method is provided for creating automated TV studio productions for various conference spaces and dedicated scenarios with virtual director assistance.

Description

[Background technology]

[0001] The present disclosure relates generally to videoconferencing and remote collaboration technologies. In particular, the present disclosure relates to enhancing videoconferencing and remote collaboration with smart cameras and smart sensors using artificial intelligence. More particularly, the present disclosure relates to an autonomous videoconferencing system with virtual director assistance and a method for creating automated television studio productions for videoconferencing spaces with virtual director assistance.

[0002] Remote collaboration and video conferencing have become a mainstay in the modern business world and society at large. In traditional video conferencing, the experience for participants is primarily static. Cameras in the meeting room are generally blind to social or spatial cues, such as speaker-listener dynamics, participants' reactions, body language, direction of attention, and degree of engagement. Thus, as a remote participant, the experience often lacks engagement. Existing hardware and software video conferencing solutions generally rely on single camera systems. Such systems are generally limited to viewing objects in a video conferencing space from one direction or one angle. Thus, it is difficult for the system to capture subtle changes in facial expressions, and gestures and postures among objects that are variously positioned in the video conferencing space. As a result, remote participants are presented with a suboptimal and less-engaging experience. In large videoconferencing spaces, it is often more difficult to capture and frame objects that are located far away from the camera, and thus it may be difficult, if not impossible, for remote participants to see or read and engage with the objects.

[0003] More recent remote collaboration and videoconferencing solutions have undergone improvements that allow remote users to adjust settings related to, among other things, their screen selection, image filtering, recording, ambient audio control, file sharing, and privacy control preferences. However, the lack of the ability of remote users to see or read the room and fully engage in real-time with all objects within the conference space remains a drawback. Thus, due to these limitations, despite the exponential growth of videoconferencing solutions in recent years, face-to-face meetings are still preferred in a variety of contexts when users are unable or unwilling to compromise their ability to actively engage in a particular meeting or conference program.

[0004] Therefore, there is a need for a smart videoconferencing solution that can capture and detect all objects and their interactions, as well as associated changes in gestures and postures, in the entire conference space. More specifically, there is a need to create a cohesive videoconferencing production, similar to a television studio production, of the entire videoconference in real time and stream such production to remote users, thereby improving their engagement. Summary of the Invention

[0005] It is therefore an object of the present disclosure to enhance a videoconferencing solution with sub-symbolic artificial intelligence, thereby enabling detection of objects and their interactions as well as associated changes in gestures and postures. It is a further object of the present disclosure to use symbolic artificial intelligence to develop a rule set that follows television production principles, and by applying such rule set, create an automated videoconferencing production for streaming in real time to a remote user.

[0006] In particular, the present disclosure provides, in one embodiment, an autonomous video conferencing system comprising a plurality of smart cameras, each of which includes an image sensor adapted to capture video within a video conferencing space and output an overview video stream and an attention video stream, where the attention video stream includes a sub-image framing a detected object in the overview video stream; an audio device adapted to capture audio signals within the video conferencing space; a vision pipeline unit adapted to receive the overview video stream and the audio signal and detect objects and poses using machine learning, where the objects include people and non-people items, and the poses include posture, orientation, gesture, and direction of the detected objects; and a video conferencing system connected to the vision pipeline unit and the audio device. a virtual director unit adapted to frame an object of interest according to a predetermined rule set, thereby enabling the video stream of interest to be updated in real time, where the predetermined rule set is adapted to create an automated television studio production from the video conferencing space; a stream selector connected to the virtual director unit and adapted to transition the video stream of interest to an updated video stream of interest; and an application programming interface connected to the virtual director unit and adapted to provide at least one connection to another smart camera of the plurality of smart cameras and one connection to a user computer.

[0007] According to one embodiment, one of the smart cameras is adapted as a main camera and each of the remaining smart cameras is adapted as a peripheral camera. The application programming interface of each peripheral camera is connected to the application programming interface of the main camera. The stream selector of each peripheral camera is further adapted to send its updated attention stream to the stream selector of the main camera, which is further adapted to select one of the updated attention streams from the smart cameras as an updated main attention stream and output the updated main attention stream to a user computer.

[0008] In another embodiment, the autonomous videoconferencing system further comprises a plurality of smart sensors disposed within the videoconferencing space and adapted to capture non-image signals within the videoconferencing space, each of the plurality of smart sensors including an application program interface adapted to be connected to the application program interface of the main camera and thereby provide input to the virtual director unit of the main camera.

[0009] In yet another embodiment, each of the plurality of smart sensors is selected from the group consisting of a touchpad, a microphone, a smartphone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor.

[0010] In a further embodiment, the audio device includes an array of microphones adapted to provide audio direction to the captured audio signals.

[0011] In another embodiment, the visual pipeline unit includes at least one hardware accelerated convolutional neural network. In yet another embodiment, the visual pipeline unit is pre-trained with a training set of video and audio data adapted to a dedicated video conferencing space. In a further embodiment, the dedicated video conferencing space is selected from the group consisting of a classroom, a workshop, a town hall, a newsroom, a boardroom, a courtroom, an interview studio, and a voting room.

[0012] In another embodiment, the predefined rule set includes a first rule for evaluating possible framings for each object in the videoconferencing space based on a first plurality of parameters, thereby determining a best framing. In yet another embodiment, the first plurality of parameters includes: (i) whether the object is speaking; (ii) the amount of time it is speaking; (iii) the direction of gaze of the object; (iv) the degree of visibility of the object in the frame; (v) the pose of the object; and (vi) which other objects are visible in the frame.

[0013] In a further embodiment, the predefined rule set further comprises a second rule for detecting a change in the videoconferencing space and triggering a frame transition based on a second plurality of parameters. In another embodiment, the second plurality of parameters comprises: (i) an object starting to speak; (ii) an object moving; (iii) an object standing up; (iv) the direction of gaze of an object changing; (v) an object showing a reaction; (vi) an object pointing to a new item in the scene; (vii) an object speaking for a predefined length of time; and (viii) a lack of meaningful reaction in other objects for a predefined length of time.

[0014] In yet another embodiment, the predefined rule set further includes a third rule for applying an appropriate shot type to each frame based on a third plurality of parameters consistent with television studio production principles. In a further embodiment, the third plurality of parameters includes: (i) an overview shot that frames substantially all objects and most of the videoconferencing space, thereby providing an overall context for the videoconferencing space; (ii) a mid-shot that frames a predefined number of objects and focuses on a person speaking, thereby highlighting an active dialogue; and (iii) a close-up shot that frames one object speaking for a predefined length of time, thereby highlighting a presenter.

[0015] In another embodiment, the third plurality of parameters further comprises: (i) a target shot framing an object of interest based on cues of the scene in the videoconferencing space, including an object in the center of the line of sight from all objects in the videoconferencing space and an item held up by the object; (ii) a near-side shot framing at least one object that is not speaking, thereby highlighting the involvement of the non-speaking object in the videoconferencing space; and (iii) a presenter shot framing a speaking object for a predefined longer length of time compared to other objects, thereby highlighting the presenter from various camera angles and compositions in the videoconferencing space. According to various embodiments, the target shot is adapted as a close shot, the near-side shot is adapted as one of a close and a mid shot, and the presenter shot is adapted as one of a close and a mid shot.

[0016] In another embodiment, the predefined rule set further includes a fourth rule for applying a virtual director's cut to the videoconferencing space based on a fourth plurality of parameters, thereby adapting it to a dedicated television production scenario. In yet another embodiment, the fourth plurality of parameters are: (i) a classroom production scenario that starts by showing a presenter and audience using an overview shot, then transitions to framing the presenter in a presenter shot for a predefined length of time, followed by switching between an audience shot showing the audience and a presenter shot showing the presenter; (ii) a workshop production scenario that starts by showing all participating objects using an overview shot, then transitions to showing active objects using a mid-shot, followed by showing objects moving within the videoconferencing space using a mid-shot, and after a predefined length of time, rotates back to showing active objects within the videoconferencing space; and (iii) a video production scenario that starts by showing all participating objects using an overview shot, then transitions to showing active objects using a mid-shot, followed by showing objects moving within the videoconferencing space using a mid-shot, and after a predefined length of time, rotates back to showing active objects within the videoconferencing space. The present invention includes a meeting room production scenario that starts with the overview shot that creates an understanding of the entire videoconferencing space with active objects, transitions after a predefined length of time to framing a group of objects with intermediate shots at sub-locations of the videoconferencing space focusing on active objects, followed by framing a talking object at the sub-location using an intermediate shot that best displays the front of the object's face, and after another predefined length of time, then switches to framing other objects in the videoconferencing space using listener shots that best display the front of the object's face, and rotating back to an overview shot featuring all objects if there are no talking objects in the videoconferencing space.

[0017] According to various embodiments, the active objects include a speaking object, a whiteboard on which content is being drawn, and items held up by an object during a presentation, and the sub-locations of the videoconferencing space include a side of a table, a stage, a bench, a podium, and a row of chairs. In a further embodiment, the meeting room production scenario is further adapted to represent a production scenario selected from the group consisting of a news broadcast or podcast, a bilateral negotiation, a court proceeding, a panel discussion, and a polling rally.

[0018] In another embodiment, the predefined rule set further includes a fifth rule for framing a clean shot of an object in the virtual meeting space. The fifth rule includes: not selecting any shot with a partially visible object; and aligning the eyes of the active object in the top third of the frame. In yet another embodiment, the fifth rule further includes adding spatial padding in the direction of the object's line of sight; and using a mid-shot to frame nearby active objects together. According to various embodiments, the nearby active objects include objects, whiteboards, display monitors, lecterns, podiums, and items being demonstrated.

[0019] In a further embodiment, the plurality of smart cameras further includes at least one peripheral smart camera disposed in a separate video conferencing space, and the predetermined rule set in the main camera is further adapted to create an automated television studio production for the combined virtual conference space.

[0020] In another embodiment, the autonomous videoconferencing system further comprises at least one smart sensor positioned within the separate videoconferencing space, the smart sensor adapted to capture non-image signals within the separate videoconferencing space and provide input to the virtual director unit of the main camera.

[0021] In yet another embodiment, the application programming interface of the main camera is adapted to receive input from the user computer thereby modifying the predetermined rule set for the virtual director unit of the main camera.

[0022] According to the present disclosure, in another embodiment, a method for creating an automated television studio production of a videoconferencing space with virtual director assistance is provided, the method comprising: capturing video in the videoconferencing space using multiple image sensors; capturing audio signals in the videoconferencing space using multiple audio devices; generating an overview video stream and an attention video stream, where the attention video stream includes sub-videos framing detected objects in the overview video stream; detecting objects and poses using machine learning from the overview stream and the audio signals, where the objects include people and non-people items, and the poses include postures, orientations, gestures, and directions of the detected objects; implementing a virtual director that includes a predefined rule set that follows television studio production principles; applying the predefined rule set to the detected objects and poses to frame objects of interest and update the attention video stream in real time; and outputting the updated attention video stream to a user computer.

[0023] In yet another embodiment, the machine learning is performed on at least one hardware accelerated convolutional neural network. In a further embodiment, the neural network is pre-trained with a training set of video and audio data adapted to a dedicated video conferencing space. In another embodiment, the dedicated video conferencing space is selected from the group consisting of a classroom, a workshop, a town hall, a newsroom, a boardroom, a courtroom, an interview studio, and a voting room.

[0024] According to various embodiments, the predefined rule set includes a first rule for evaluating possible framings for each object in the videoconferencing space based on a first plurality of parameters, thereby determining a best framing, a second rule for detecting changes in the videoconferencing space based on a second plurality of parameters and triggering frame transitions, a third rule for applying an appropriate shot type to each frame based on a third plurality of parameters that correspond to television studio production principles, a fourth rule for applying a virtual director's cut to the videoconferencing space based on a fourth plurality of parameters, thereby adapting it to a dedicated television production scenario, and a fifth rule for framing a clean shot for objects in the virtual conference space based on a fifth plurality of parameters.

[0025] In another embodiment, the shot types are selected from the group consisting of overview shot, mid shot, close up shot, subject shot, audience shot, and presenter shot. In yet another embodiment, the specialized television production scenario comprises a classroom, a workshop, a meeting room, a broadcast, a bilateral negotiation, a court proceeding, a panel discussion, and a polling session.

[0026] In a further embodiment, the method for creating an automated television studio production further comprises capturing non-image signals within the videoconferencing space using a plurality of smart sensors, each of which includes an application program interface connected to the virtual director to provide input to the virtual director. In various embodiments, each of the plurality of smart sensors is selected from the group consisting of a touchpad, a microphone, a smartphone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a diagram of an autonomous videoconferencing system according to one embodiment.

[0028] [Diagram 2] FIG. 2 is a diagram of an overview video stream and a focus video stream according to one embodiment.

[0029] [Diagram 3] 4A-4C are diagrams of a full shot, a mid shot, and a close shot according to one of the rules of a predefined rule set in one embodiment.

[0030] [Figure 4] FIG. 2 illustrates the framing of a clean shot according to one of the rules of a predefined rule set in one embodiment.

[0031] [Diagram 5] FIG. 2 is a diagram of alignment for framing a clean shot according to one of the rules of a predefined rule set in one embodiment.

[0032] [Figure 6]1A-1C are diagrams of two example small videoconferencing spaces for dedicated scenarios in which autonomous videoconferencing systems are deployed, according to one embodiment.

[0033] [Figure 7] 1A-1C are diagrams of two example medium-sized videoconferencing spaces for a dedicated scenario in which an autonomous videoconferencing system is deployed, according to one embodiment.

[0034] [Figure 8] 1A-1C are diagrams of two examples of large videoconferencing spaces for dedicated scenarios in which autonomous videoconferencing systems are deployed, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] The videoconferencing system and method of the present disclosure are enhanced by multiple smart cameras and smart sensors that use sub-symbolic and symbolic artificial intelligence. In various embodiments, the autonomous videoconferencing system of the present disclosure includes multiple smart cameras that are optionally coupled with multiple smart sensors located in the videoconferencing space. Each smart camera is equipped with a vision pipeline supported by machine learning to detect objects and poses and identify speaker-listener dynamics, and a virtual director that is adapted to apply a predefined set of rules that match television studio production principles. At the same time, methods are provided for creating automated television studio productions for various conference spaces and dedicated scenarios with virtual director assistance. Multiple Smart Cameras and Sensors

[0036] Referring to FIG. 1, an autonomous video conferencing (AVC) system according to one embodiment includes a plurality of smart cameras (100, 101) and, optionally, one or more smart sensors (102). One of the plurality of smart cameras is adapted as a main camera (100), and the rest of the plurality of smart cameras are adapted as peripheral cameras (101). Each of the plurality of smart cameras includes an image sensor (201, 301), an audio device (204, 304), a vision pipeline unit or vision pipeline (202, 302), a virtual director unit or virtual director (203, 303), a stream selector (205, 305), and an application programming interface (API) (206, 306). Each of the one or more smart sensors includes an API (401) that can be connected to the API of the smart camera. According to one embodiment, the API of the main camera (100) is connected to the API of each peripheral camera (101) and to the API of each smart sensor (102). The API of the main camera is further adapted to provide a connection to a user computer (103).

[0037] The smart cameras and one or more smart sensors in the AVC system are connected via Ethernet, other local area networks, or wireless networks in various embodiments. According to one embodiment, the main camera and peripheral cameras are variously positioned within the videoconferencing space to provide effective coverage of the videoconferencing space. In another embodiment, the smart sensors are activated and strategically positioned within the videoconferencing space to capture non-image signals and provide input to the main camera of the AVC system. The smart sensors of the present disclosure include, in various embodiments, a touchpad, a microphone, a smartphone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor.

[0038] In alternative embodiments, one or more peripheral cameras and smart sensors of the AVC system are located in a separate videoconferencing space as a secondary space for videoconferencing. These peripheral cameras and smart sensors are networked with the main camera and adapted to provide image and non-image inputs from the secondary space to the main camera. Thus, the AVC system of these alternative embodiments is further adapted to generate an automated television studio production for the combined videoconferencing space based on inputs from all cameras and smart sensors in both spaces.

[0039] The smart cameras of the AVC system are adapted with various angles of view in various embodiments. For example, if the videoconferencing space is small and the AVC system has few cameras, the smart cameras may have a wide field of view, for example, covering approximately 150 degrees. On the other hand, if the videoconferencing space is large and the AVC system has many cameras, the smart cameras may have a narrower field of view, for example, covering approximately 90 degrees. In another embodiment, the AVC system is equipped with smart cameras with various angles of view, which allows optimal coverage of the videoconferencing space. In a further embodiment, the image sensor (201, 301) is adapted to zoom up to 10 times, allowing close-up images of objects at the back of the videoconferencing space. In an alternative embodiment, one or more smart cameras in the AVC system are adapted to capture content on or around objects that are non-person items in the videoconferencing space, such as whiteboards, TV displays, posters, and podiums. These cameras are smaller and may be positioned differently than other smart cameras in the AVC system, and may be ceiling-mounted to provide effective coverage of targeted content.

[0040] In a further embodiment, the audio device (204, 304) in the smart camera is a microphone array adapted to capture audio signals from various positions around the camera. Using the signals from the various microphones, the smart camera can determine the direction of audio (DOA) and identify whether silence exists at a certain position or direction. This information is also made available to the visual pipeline (202, 302) and the virtual director (203, 303) of the AVC system. In a further alternative embodiment, a computing device with high computing power is connected to the AVC system through an Ethernet switch and adapted to provide additional computing power to the AVC system. It has one or more high performance CPUs and GPUs and runs a portion of the visual pipeline for the main camera and any designated peripheral cameras.

[0041] Therefore, the inclusion and configuration of multiple smart cameras and smart sensors within an AVC system allows for effective and robust coverage of a variety of videoconferencing spaces and scenarios. By distributing multiple smart cameras within a videoconferencing space that work together to frame objects within the conference from various camera angles and zoom levels, the AVC system of the present disclosure creates a more holistic, natural, and engaging experience for all participants, including remote users. Visual Pipeline; Overview Stream & Attention Stream

[0042] Referring to Figure 2, for each smart camera in the AVC system, there are two video streams internally: an overview stream and an attention stream. The overview stream shows the entire scene, which is consumed by the vision pipeline (202, 302) as shown in Figure 1. The attention stream is a high-resolution stream that frames objects of interest according to the activity in the videoconferencing space over time. Video settings are then applied under the direction of the virtual director (203, 303), which transitions the attention stream to an updated attention stream, as discussed in detail below.

[0043] The vision pipeline of the present disclosure is adapted to process the incoming overview stream and audio signals and detect objects and poses using machine learning. In various embodiments, the objects include people and non-people items, and the poses include body posture, orientation, gestures, and direction. The vision pipeline in one embodiment includes one or more hardware-accelerated programmable convolutional neural networks that employ pre-trained weights to enable detection of certain characteristics of objects in the field of view. For example, the vision pipeline detects where objects are in the smart camera's field of view, their degree of visibility in the field of view, whether they are speaking, their facial expressions, their body posture, and their head pose. The vision pipeline also tracks each object over time and determines where the objects were previously in the field of view, whether they are moving, and which direction they are heading.

[0044] One advantage of the vision pipeline, according to various embodiments, leveraging sub-symbolic artificial intelligence to detect objects and their activities and interactions is that these convolutional neural networks are trained to be unbiased based on characteristics such as gender, age, race, scene, lighting, and size, allowing the AVC system to create a more accurate and natural video stream production for the entire scene and all objects in the videoconferencing space.

[0045] The visual pipeline of various embodiments is adapted to run on a GPU or other dedicated chipset with hardware accelerators for the relevant mathematical operations in its convolutional neural network. In an alternative embodiment, the visual pipeline runs on the CPU power available in the AVC system. Further optimization of the visual pipeline to fit the hardware chipset in certain embodiments is done by replacing the mathematical operations in its convolutional neural network architecture with equivalent mathematical operations supported by the chipset. Therefore, the dedicated hardware support of the visual pipeline allows it to perform object and pose detection with high frequency and fast processing time, which further enables the AVC system to react quickly and on demand to changes in the field of view of its smart camera.

[0046] In a particular embodiment, the visual pipeline is pre-trained by running thousands of images and videos related to the scene and detection target for the videoconferencing space. During the training, the visual pipeline is evaluated with a loss function that measures how well it performs a particular detection. Feedback from the loss function is then used to adjust the weights and parameters of the visual pipeline until it performs the particular detection at a predefined level of achievement. In one embodiment, the visual pipeline is further pre-trained and fine-tuned with a training set of video and audio data that is adapted to a dedicated videoconferencing space for a dedicated scenario. For example, the visual pipeline may be fine-tuned for a classroom, workshop, town hall, newsroom, boardroom, courtroom, interview studio, or voting room to support a dedicated scenario, such as a lecture, interview, news broadcast or podcast, courtroom proceedings, workshop, bilateral negotiation, or voting rally.

[0047] The vision pipeline is further adapted to collect and process audio signals from microphones or other audio devices in the AVC system in various embodiments. It is able to distinguish between sounds, including whether an object has made the sound louder or quieter, depending on what is happening in the videoconferencing space. In one embodiment, the vision pipeline is adapted to classify the topic of conversation based on the audio signals. Sounds that do not belong to an object are classified as artificial sounds and may be attributed to other sources, e.g., loudspeakers. The sound classifications and features are combined by the vision pipeline with other information and knowledge it detects and gathers from the image data regarding relevant objects and their activities and interactions, thereby generating a comprehensive understanding of the videoconferencing space and all detected objects. The vision pipeline makes this corpus of comprehensive understanding available to the virtual director, which is further responsible for selecting the best shots and creating an automated television studio production from the videoconferencing space.

[0048] Virtual Director; Predefined Ruleset As discussed above, the virtual director (203, 303) of the AVC system is connected to the visual pipeline (202, 302) and audio devices (204, 304) as shown in FIG. 1 and is adapted to frame objects of interest according to a predefined set of rules, thereby allowing the attention stream to be updated in real time. The predefined set of rules is formulated and implemented for the virtual director in accordance with television studio production principles, as discussed in detail below. The transition of the attention stream to the updated attention stream is performed by the stream selector (205, 305) at the direction of the virtual director in each smart camera.

[0049] The multiple smart cameras of the AVC system seamlessly cooperate to generate and update attention streams for various positions and target objects in the videoconferencing space. The main camera's virtual director (203) is connected to each peripheral camera's virtual director (303) through their respective APIs (206, 306) as shown in FIG. 1. The main camera's stream selector (205) is connected to each peripheral camera's stream selector (305) and is adapted to consume attention streams from the main camera and all peripheral cameras. The main camera's stream selector (205) is further responsible for selecting one of the updated attention streams from all cameras in the AVC system as the updated main attention stream at the direction of the main camera's virtual director (203). This updated main attention stream is the stream output available to the user computer as shown in FIG. 1. Thus, the Main Camera Virtual Director (203) is the brain or command center of the AVC system, responsible for creating an automated TV studio production of the videoconferencing space based on input from all cameras and any smart sensors deployed within the AVC system.

[0050] The Virtual Director of the present disclosure is a software component that leverages rule-based symbolic artificial intelligence to optimize decision making in various embodiments. The Virtual Director takes input from the visual pipeline and determines which frames from which cameras in the AVC system should be selected and ultimately streamed to the user. In certain embodiments, the Virtual Director achieves this by evaluating possible framings for each object in the videoconferencing space and their activity over time. For each object captured by a particular smart camera in the AVC system, for example, the Virtual Director evaluates various crops of images in which the object is visible to find the best frame for that object in the context of their activity and interaction. The Virtual Director then determines the appropriate video settings to transition the attention stream of the particular camera to the selected best frame. The transition and update of the attention stream using these video settings is then performed by the stream selector at the direction of the Virtual Director.

[0051] In certain embodiments, the main camera's virtual director also gets input from one or more smart sensors deployed in the AVC system through their respective APIs. Input from the smart sensors includes general non-image signals or cues about the videoconferencing space and all objects therein, such as object position, movement, and physiological or biometric characteristics.

[0052] The API of the AVC system is adapted to send messages between various components via an internal network bus. These messages include information about the status of each smart camera, such as whether it is connected, what type of software it is running, current health status, etc. The API also communicates what the camera detects from the image data, such as where objects are detected in the image, where they are located in the conference space, and other information detected by the vision pipeline. The API also communicates the video settings that are being applied to the attention stream, such as image characteristics, color, brightness, etc. Furthermore, the API is adapted to communicate Virtual Director parameters, which allows the AVC system to automatically set and adjust Virtual Director rule sets and related parameters for all of its component cameras, and allows users of the AVC system to personalize their Virtual Director experience by modifying certain parameters and rules for the predefined rule sets.

[0053] As discussed above, the virtual director of the present disclosure utilizes rule-based decision making that leverages symbolic artificial intelligence. The predefined rule set is formulated according to television studio production principles in various embodiments. This allows the AVC system to generate an automated television studio production of a conference experience that resembles a real-life expert-directed television production. In certain embodiments, the predefined rule set in each camera of the AVC system may be adapted differently for its location and the target object or content in the video conference space. In one embodiment, as shown in FIG. 1, the predefined rule set applied by the main camera's virtual director (203) includes more rules than the predefined rule set applied by the peripheral camera's virtual director (303). In an alternative embodiment, the user is provided with the option to modify or tune certain rules and parameters of the predefined rule set in the AVC system through the main camera's API, which is connected to the main camera's virtual director unit (203) as shown in FIG. 1.

[0054] According to one embodiment, the predefined rule set includes a first rule for evaluating possible framings for each object in the videoconferencing space based on a first plurality of parameters. The first rule defines a best frame for each object. The first plurality of parameters in one embodiment includes: (i) whether the object is speaking; (ii) the amount of time the object is speaking; (iii) the direction of the object's gaze; (iv) the degree of visibility of the object in the frame; (v) the pose of the object; and (vi) which other objects are visible in the frame.

[0055] The predefined rule set includes a second rule in another embodiment for detecting a change in the videoconferencing space based on a second plurality of parameters. The second rule triggers a frame transition. The second plurality of parameters in one embodiment includes: (i) the object starts speaking; (ii) the object moves; (iii) the object stands up; (iv) the object changes its gaze direction; (v) the object shows a reaction; (vi) the object points to a new item in the scene; (vii) the object speaks for a predefined length of time; and (viii) the lack of a meaningful reaction in other objects for a predefined length of time.

[0056] The predefined rule set includes a third rule in another embodiment for applying an appropriate shot type to each frame based on a third plurality of parameters. The third plurality of parameters in one embodiment includes: (i) an overview shot that frames substantially all objects and most of the videoconferencing space, thereby providing an overall context for the videoconferencing space, see the top frame of FIG. 3; (ii) a mid-shot that frames a predefined number of objects and focuses on the person speaking, thereby highlighting the active dialogue, see the middle frame of FIG. 3; and (iii) a close-up shot that frames one object speaking for a predefined length of time, thereby highlighting the presenter, see the bottom frame of FIG. 3.

[0057] In another embodiment, the parameters of the third rule further include: (i) a target shot that frames an object of interest based on scene cues in the videoconferencing space, including an object in the center of line of sight from all objects in the videoconferencing space and an item held up by the object; (ii) a near-side shot that frames at least one object that is not speaking, thereby featuring the engagement of the non-speaking objects in the videoconferencing space; and (iii) a presenter shot that frames the object that is speaking for the longest length of time relative to other objects, thereby featuring the presenter from various camera angles and compositions in the videoconferencing space. In various embodiments, the target shot is adapted as a close shot, the near-side shot is adapted as a close shot or a mid shot, and the presenter shot is adapted as a close shot or a mid shot.

[0058] The predefined rule set includes a fourth rule in another embodiment for applying a virtual director's cut to the video conferencing space based on a fourth plurality of parameters. The fourth rule tailors the video settings to a dedicated television production scenario. The fourth plurality of parameters in one embodiment includes a classroom production scenario, a workshop production scenario, and a meeting room production scenario. The meeting room production scenario is further adapted for news broadcasts or podcasts, bilateral negotiations, court proceedings, panel discussions, and polling rallies in certain embodiments.

[0059] The predefined rule set includes a fifth rule in a further embodiment for framing a clean shot of an object in the virtual meeting space. The fifth rule in one embodiment includes not selecting any shot with a partially visible object. With reference to FIG. 4, for example, the left frame is a clean shot and the right frame is not a clean shot and is not selected in the AVC system under the fifth rule. The fifth rule in another embodiment includes aligning the eyes of an active object in the top third of the frame. With reference to FIG. 5, for example, each of the images (top, middle, bottom) is optimally aligned with the eyes of an active object in the top third of the frame. These shots are the clean shots framed and selected in the AVC system under the fifth rule.

[0060] In yet another embodiment, the fifth rule includes adding spatial padding in the direction of the object's line of sight and using a mid-shot to frame nearby active objects together, which in various embodiments include objects, whiteboards, display monitors, lecterns, podiums, and items in a demonstration. Dedicated scenario

[0061] The AVC system of the present disclosure, in various embodiments, is adapted for a variety of dedicated video conferencing spaces for dedicated video conferencing scenarios. The vision pipeline units of the smart cameras in the AVC system are pre-trained with training sets of video and audio data that are adapted for dedicated video conferencing spaces, including classrooms, workshops, town halls, newsrooms, boardrooms, courtrooms, interview studios, and voting rooms, according to certain embodiments.

[0062] As discussed above, among the predefined rule sets implemented and applied by the virtual director unit of the AVC system is a fourth rule for creating a virtual director's cut based on a fourth plurality of parameters. This fourth rule for the virtual director's cut is designed to tailor the videoconferencing space to a dedicated television production scenario. The fourth plurality of parameters in one embodiment includes a classroom production scenario, a workshop production scenario, and a meeting room production scenario.

[0063] In one embodiment, the classroom scenario begins by showing the presenter and audience using an overview shot, then transitions to framing the presenter in a presenter shot for a predefined amount of time, and then changes to switching between audience shots showing the audience and presenter shots showing the presenter.

[0064] In one embodiment, the workshop production scenario begins by showing all participating objects using an overview shot, then transitions to showing active objects using medium shots, then changes to showing objects moving within the videoconferencing space using medium shots, and finally rotates back to showing active objects in the videoconferencing space after a predefined amount of time.

[0065] The meeting room production scenario in one embodiment starts with an overview shot that creates an understanding of the entire videoconferencing space with all visible objects, transitions after a predefined length of time to framing a group of objects with mid-shots in a sub-location of the videoconferencing space focusing on the active object, then changes to framing the talking object in the sub-location using a mid-shot that best displays the front of the object's face, after another predefined length of time, then switches to framing other objects in the videoconferencing space using listener shots that best display the front of the object's face, and finally rotates back to an overview shot featuring all objects when there is no talking object in the videoconferencing space. The active object in various embodiments represents a talking object, a whiteboard on which its contents are being drawn, or an item held up by an object during a demonstration. The sub-location of the videoconferencing space in various embodiments represents the side of a table, a stage, a bench, a podium, or a row of chairs.

[0066] Meeting room production scenarios in certain embodiments are further adapted to represent news broadcasts or podcasts, interviews, executive meetings, bilateral negotiations, court proceedings, panel discussions, and polling sessions.

[0067] For various dedicated scenarios, the main camera and peripheral cameras are strategically and variously positioned within the videoconferencing space to provide effective coverage of the space. One or more optional smart sensors are additionally distributed within the videoconferencing space with a connection to the main camera to provide input to the virtual director unit of the main camera. Examples of the AVC system of the present disclosure deployed for dedicated scenarios are shown in Figures 6 to 8. In these figures, each dot represents a smart camera or smart sensor of the AVC system, and each circle represents an object including a person or a non-person item, such as a chair or demonstration material. The small rectangles along the sides of each of these figures represent a TV, whiteboard, poster, or projection display (601, 602, 701, 702, 703, 704, 801, 802). The square or rectangle in the middle of the figure represents a table.

[0068] Referring to Figure 6, both the top and bottom depict small videoconferencing spaces in which AVC systems are deployed. The top configuration represents a small meeting and panel discussion scenario in one embodiment. The bottom configuration represents a news broadcast or podcast and interview scenario in one embodiment.

[0069] 7, both the top and bottom depict a medium-sized videoconferencing space in which an AVC system is deployed. The top configuration represents an executive meeting scenario in one embodiment. The bottom configuration represents a bilateral negotiation in one embodiment.

[0070] Referring to Figure 8, both the top and bottom depict a large videoconferencing space in which an AVC system is deployed. The top configuration represents a workshop scenario in one embodiment. The bottom configuration represents a classroom scenario in one embodiment.

[0071] The description of the various embodiments, including the figures and examples, is intended to be illustrative and not limiting of the invention and its various embodiments.

Claims

1. 1. An autonomous videoconferencing system comprising a plurality of smart cameras, each of the plurality of smart cameras comprising: an image sensor adapted to capture video within the video conferencing space and output an overview video stream and an attention video stream, where the attention video stream includes a sub-image framing a detected object in the overview video stream; an audio device adapted to capture an audio signal within the videoconferencing space; a vision pipeline unit adapted to receive the overview video stream and the audio signal and to detect objects and poses using machine learning, where the objects include people and non-people items, and the poses include posture, orientation, gestures, and direction of the detected objects; a virtual director unit, connected to the visual pipeline unit and the audio device, adapted to frame an object of interest according to a predefined set of rules, thereby enabling the focus video stream to be updated in real time, where the predefined set of rules is adapted to create an automated television studio production from the video conferencing space; a stream selector coupled to the virtual director unit and adapted to transition the video stream of interest to an updated video stream of interest; and an application programming interface coupled to the virtual director unit and adapted to provide at least one connection to another smart camera of the plurality of smart cameras and one connection to a user computer; having one of the plurality of smart cameras is adapted as a main camera and each of the remaining plurality of smart cameras is adapted as a peripheral camera, the application programming interface of each peripheral camera is connected to the application programming interface of the main camera, the stream selector of each peripheral camera is further adapted to send its updated attention stream to the stream selector of the main camera, the stream selector of the main camera is further adapted to select one of the updated attention streams from the plurality of smart cameras as an updated main attention stream and output the updated main attention stream to a user computer; Autonomous videoconferencing system.

2. 2. The autonomous video conferencing system of claim 1, further comprising: a plurality of smart sensors disposed within the video conferencing space and adapted to capture non-image signals within the video conferencing space, each of the plurality of smart sensors including an application program interface connected to the application program interface of the main camera, thereby adapted to provide input to the virtual director unit of the main camera.

3. 3. The autonomous videoconferencing system of claim 2, wherein each of the plurality of smart sensors is selected from the group consisting of a touchpad, a microphone, a smartphone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor.

4. 4. The autonomous videoconferencing system of claim 1, wherein the audio device includes an array of microphones adapted to provide audio direction to the captured audio signals.

5. 4. The autonomous videoconferencing system of claim 1, wherein the visual pipeline unit includes at least one hardware accelerated convolutional neural network.

6. The autonomous videoconferencing system of claim 5 , wherein the visual pipeline unit is pre-trained with a training set of video and audio data adapted for a dedicated videoconferencing space.

7. 7. The autonomous videoconferencing system of claim 6, wherein the dedicated videoconferencing space is selected from the group consisting of a classroom, a workshop, a town hall, a newsroom, a boardroom, a courtroom, an interview studio, and a voting room.

8. 4. The autonomous video conferencing system of claim 1, wherein the predetermined rule set includes a first rule for evaluating possible framings for each object in the video conferencing space based on a first plurality of parameters, thereby determining a best framing.

9. 9. The autonomous video conferencing system of claim 8, wherein the first plurality of parameters includes: (i) whether the object is speaking; (ii) the length of time the object is speaking; (iii) the direction of the object's gaze; (iv) the degree of visibility of the object in the frame; (v) the pose of the object; and (vi) which other objects are visible in the frame.

10. 10. The autonomous videoconferencing system of claim 8, wherein the predetermined rule set further comprises a second rule for detecting a change in the videoconferencing space based on a second plurality of parameters and triggering a frame transition.

11. The second plurality of parameters include: (i) the object begins to speak; (ii) the object moves; (iii) the object stands up; (iv) the direction of the object's gaze changes; (v) the object shows a reaction; (vi) the object points to a new item in the scene; (vii) the object speaks for a predefined length of time; and (viii) the lack of a meaningful reaction in other objects for a predefined length of time.

11. The autonomous video conferencing system of claim 10, comprising:

12. 11. The autonomous videoconferencing system of claim 10, wherein the predetermined rule set further comprises a third rule for applying an appropriate shot type to each frame based on a third plurality of parameters consistent with television studio production principles.

13. The third plurality of parameters include: (i) an overview shot that frames substantially all objects and most of the videoconferencing space, thereby providing an overall context for the videoconferencing space; (ii) a medium shot that frames a predefined number of objects and focuses on a person speaking, thereby highlighting an active dialogue; and (iii) a close-up shot that frames one object speaking for a predefined length of time, thereby highlighting a presenter.

13. The autonomous videoconferencing system of claim 12.

14. 14. The autonomous videoconferencing system of claim 13, wherein the third plurality of parameters further includes: (i) a target shot that frames an object of interest based on scene cues in the videoconferencing space, including an object at the center of line of sight from all objects in the videoconferencing space and an item held up by the object; (ii) a listener shot that frames at least one object that is not speaking, thereby featuring engagement of non-speaking objects in the videoconferencing space; and (iii) a presenter shot that frames an object that is speaking for the longest length of time compared to other objects, thereby featuring the presenter from various camera angles and compositions in the videoconferencing space, wherein the target shot is adapted as a close shot, the listener shot is adapted as one of a close and a medium shot, and the presenter shot is adapted as one of a close and a medium shot.

15. 13. The autonomous videoconferencing system of claim 12, wherein the predetermined rule set further includes a fourth rule for applying a virtual director's cut to the videoconferencing space based on a fourth plurality of parameters, thereby tailoring it to a dedicated television production scenario.

16. The fourth plurality of parameters is: (i) a classroom production scenario that starts by showing a presenter and audience using an overview shot, then transitions to framing the presenter in a presenter shot for a predefined length of time, followed by switching between audience shots showing the audience and presenter shots showing the presenter; (ii) a workshop production scenario that starts by showing all participating objects using an overview shot, then transitions to showing active objects using a medium shot, followed by showing objects moving within the videoconferencing space using a medium shot, and after a predefined length of time rotates back to showing active objects within the videoconferencing space; and (iii) the above mentioned videoconferencing scenario with all visible objects. a meeting room production scenario starting with the overview shot that creates an understanding of the entire videoconferencing space, transitioning after a predefined length of time to framing a group of objects with intermediate shots at a sub-location of the videoconferencing space focusing on an active object, followed by framing a talking object at the sub-location using an intermediate shot that best displays the front of the object's face, and after another predefined length of time then switching to framing other objects in the videoconferencing space using listener shots that best display the front of the object's face, rotating back to an overview shot featuring all objects if there is no talking object in the videoconferencing space, the active objects include an object that is speaking, a whiteboard on which content is being drawn, and items held up by an object during a presentation, and the sub-locations of the videoconferencing space include sides of a table, a stage, a bench, a podium, and a row of chairs; 16. The autonomous videoconferencing system of claim 15.

17. 20. The autonomous video conferencing system of claim 16, wherein the meeting room production scenario is further adapted to represent a production scenario selected from the group consisting of a news broadcast or podcast, a bilateral negotiation, a court proceeding, a panel discussion, and a polling meeting.

18. 16. The autonomous videoconferencing system of claim 15, wherein the predetermined rule set further includes a fifth rule for framing a clean shot of an object within the virtual conference space, the fifth rule including: not selecting any shot with a partially visible object; and aligning the eyes of an active object in the top third of the frame.

19. 20. The autonomous video conferencing system of claim 18, wherein the fifth rule further comprises: adding spatial padding in the direction of an object's line of sight; and using a mid-shot to frame nearby active objects together, the nearby active objects including objects, whiteboards, display monitors, lecterns, podiums, and items in a demonstration.

20. 4. The autonomous video conferencing system of claim 1, wherein the plurality of smart cameras further includes at least one peripheral smart camera disposed in a separate video conferencing space, and the predetermined rule set in the main camera is further adapted to create an automated television studio production for the combined virtual conference space.

21. 21. The autonomous video conferencing system of claim 20, further comprising at least one smart sensor positioned within the separate video conferencing space, the smart sensor adapted to capture non-image signals within the separate video conferencing space and provide input to the virtual director unit of the main camera.

22. 4. The autonomous video conferencing system of claim 1, wherein the application programming interface of the main camera is adapted to receive input from the user computer and thereby modify the predetermined rule set for the virtual director unit of the main camera.

23. 1. A method for creating an automated television studio production of a videoconferencing space with virtual director assistance, comprising: capturing video within the videoconferencing space using multiple image sensors; capturing audio signals within the videoconferencing space using multiple audio devices; generating an overview video stream and an attention video stream, where the attention video stream includes sub-images framing detected objects in the overview video stream; detecting objects and poses using machine learning from the overview video stream and the audio signal, where the objects include people and non-people items and the poses include posture, orientation, gestures, and direction of the detected objects; implementing a virtual director that includes a predetermined rule set that follows television studio production principles; applying the predetermined rule set to the detected objects and poses to thereby frame objects of interest and update the attention video stream in real time; and outputting the updated attention video stream to a user computer.

24. 24. The method of claim 23, wherein the machine learning is performed on at least one hardware accelerated convolutional neural network.

25. 25. The method of claim 24, wherein the neural network is pre-trained with a training set of video and audio data adapted to a dedicated videoconferencing space.

26. 26. The method of claim 25, wherein the dedicated video conferencing space is selected from the group consisting of a classroom, a workshop, a town hall, a newsroom, a boardroom, a courtroom, an interview studio, and a voting room.

27. 27. The method of claim 23, wherein the predetermined rule set includes a first rule for evaluating possible framings for each object in the videoconferencing space based on a first plurality of parameters, thereby determining a best framing.

28. 30. The method of claim 27, wherein the predetermined rule set further comprises a second rule for detecting a change in the videoconferencing space based on a second plurality of parameters and triggering a frame transition.

29. 30. The method of claim 28, wherein the predetermined rule set further comprises a third rule for applying an appropriate shot type to each frame based on a third plurality of parameters consistent with television studio production principles.

30. 30. The method of claim 29, wherein the shot types are selected from the group consisting of overview shot, mid shot, close up shot, object shot, near-field shot, and presenter shot.

31. 31. The method of claim 30, wherein the predetermined rule set further comprises a fourth rule for applying a virtual director's cut to the videoconferencing space based on a fourth plurality of parameters, thereby tailoring it to a dedicated television production scenario.

32. 32. The method of claim 31, wherein the specialized television production scenarios include classrooms, workshops, meeting rooms, broadcasts, bilateral negotiations, court proceedings, panel discussions, and polling sessions.

33. 33. The method of claim 32, wherein the predetermined rule set further comprises a fifth rule for framing a clean shot of an object in the virtual meeting space based on a fifth plurality of parameters.

34. 27. The method of any one of claims 23 to 26, further comprising using a plurality of smart sensors to capture non-image signals within the videoconferencing space, each of the plurality of smart sensors including an application program interface connected to the Virtual Director thereby providing input to the Virtual Director.

35. 35. The method of claim 34, wherein each of the plurality of smart sensors is selected from the group consisting of a touchpad, a microphone, a smartphone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor.

Images