Autonomous video conferencing system with virtual director assistance

Through the combination of smart cameras and sensors, the video stream is updated in real time, which solves the problem that traditional video conferencing systems are difficult to capture the objects and their changes in the conference space, and improves the sense of participation and experience quality of remote participants.

CN119096270APending Publication Date: 2024-12-06HUDDLY INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280092229.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-02-23
Filing Date
2022-04-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional video conferencing systems have difficulty capturing all objects in the conference space and their interactions and posture changes, resulting in a lack of participation in remote participants experience, especially in large conference spaces that are difficult to effectively capture and display objects away from the camera.

Method used

Using multiple smart cameras and smart sensors, combining machine learning and virtual director technology, focus video streams in real time through pre-determined rule sets, simulate TV studio production, providing a rich video conferencing experience.

Benefits of technology

Real-time detection and display of all objects in the video conferencing space is achieved, which improves the participation and experience quality of remote participants, and provides a cohesion effect similar to that of TV studios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096270A_ABST
    Figure CN119096270A_ABST
Patent Text Reader

Abstract

Systems and methods are provided to power video conferencing and remote collaboration with sub-symbol and symbol artificial intelligence. An autonomous video conferencing system of the present disclosure includes a master smart camera and a plurality of peripheral smart cameras, optionally coupled with one or more smart sensors. Each smart camera is equipped with: a visual pipeline supported by machine learning for detecting objects and their interactions and related changes in gestures and gestures; and a virtual director adapted to apply a predetermined set of rules consistent with the television studio production principle. The main camera is adapted to select and update the focused video stream in real time under the guidance of its virtual director and stream the updated focused stream to the user computer. Methods are provided for utilizing virtual directors to assist in creating automatic television studio fabrication for various conference spaces and dedicated scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to video conferencing and remote collaboration technologies. Specifically, the present disclosure relates to using artificial intelligence to power video conferencing and remote collaboration using smart cameras and smart sensors. More specifically, the present disclosure relates to an autonomous video conferencing system with virtual director assistance and a method for creating an automated television studio production for a video conferencing space with virtual director assistance. Background Art

[0002] Remote collaboration and video conferencing have become the backbone of the modern business world and society in general. In traditional video conferencing, the experience of participants is largely static. Cameras in conference rooms are often unaware of social or spatial cues such as speaker-listener dynamics, participant reactions, body language, attention direction, and engagement levels. As a result, as a remote participant, the experience is often lacking in engagement. Existing hardware and software video conferencing solutions often rely on a single camera system. Such systems are often limited to displaying objects in the video conferencing space from one direction or angle. It is therefore challenging for the system to capture subtle changes in facial expressions and gestures and postures between objects that are differently located in the video conferencing space. As a result, a suboptimal and low-engagement experience is presented to remote participants. In large video conferencing spaces, it can often be difficult to additionally capture and frame objects that are far from the camera, so it is also difficult (if not impossible) for remote participants to see or read objects and interact with them.

[0003] More recent remote collaboration and video conferencing solutions have seen improvements that enable remote users to adjust settings for their screen selection, image filtering, recording, ambient audio control, file sharing, and privacy control preferences, among others. However, the remote user's lack of ability to see or read the room and interact with all objects in the meeting space in full real-time remains a drawback. Therefore, due to these limitations, despite the exponential growth of video conferencing solutions in recent years, face-to-face meetings are still preferred in a variety of environments if users are unable or unwilling to compromise on their ability to actively participate in a particular meeting or conference proceedings.

[0004] Therefore, there is a need for intelligent video conferencing solutions that are able to capture and detect all objects and their interactions and associated changes in gestures and postures throughout the conference space. More specifically, there is a need to create a cohesive video conferencing production similar to a TV studio production for the entire video conference in real time and stream this production to remote users, thereby improving remote user engagement. Summary of the invention

[0005] Therefore, an object of the present disclosure is to provide a sub-symbolic artificial intelligence for a video conferencing solution, thereby enabling detection of objects and their interactions and related changes in gestures and postures. Another object of the present disclosure is to develop a rule set that complies with the principles of television production using symbolic artificial intelligence and to create an automated video conference production for real-time streaming to remote users by applying such a rule set.

[0006] In particular, according to the present disclosure, in one embodiment, an autonomous video conferencing system is provided, comprising a plurality of smart cameras. Each of the plurality of smart cameras comprises: an image sensor adapted to capture video images in a video conferencing space and output an overview video stream and a focus video stream, wherein the focus video stream comprises a sub-video image that frames a detected object within the overview video stream; an audio device adapted to capture an audio signal in the video conferencing space; a visual pipeline unit adapted to receive the overview video stream and the audio signal and detect objects and gestures by means of machine learning, wherein the objects comprise human and non-human items and the gestures comprise the position, orientation, gesture and direction of the detected objects; a virtual director unit connected to the visual pipeline unit and the audio device and adapted to frame objects of interest according to a predetermined set of rules, thereby allowing real-time updating of the focus video stream, wherein the predetermined set of rules is adapted to create an automated television studio production according to the video conferencing space; a stream selector connected to the virtual director unit and adapted to convert the focus video stream into an updated focus video stream; and an application programming interface connected to the virtual director unit and adapted to provide at least one connection to another smart camera of the plurality of smart cameras and one connection to a user computer,

[0007] According to one embodiment, one of the plurality of smart cameras is adapted to be a main camera, and each of the remaining smart cameras of the plurality of smart cameras is adapted to be a peripheral camera. An application programming interface of each peripheral camera is connected to an application programming interface of the main camera. The stream selector of each peripheral camera is further adapted to transmit its updated focus stream to the stream selector of the main camera, and the stream selector of the main camera is further adapted to select one of the updated focus streams from the plurality of smart cameras as an updated main focus stream, and output the updated main focus stream to a user computer.

[0008] In another embodiment, the autonomous video conferencing system further comprises a plurality of smart sensors placed in the video conferencing space and adapted to capture non-image signals in the video conferencing space. Each of the plurality of smart sensors comprises an application programming interface adapted to be connected to an application programming interface of the main camera, thereby providing input to the virtual director unit of the main camera.

[0009] In yet another embodiment, each of the plurality of smart sensors is selected from the group consisting of a touchpad, a microphone, a smart phone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor.

[0010] In another embodiment, the audio device comprises a microphone array adapted to provide directionality of audio using the captured audio signal.

[0011] In another embodiment, the visual pipeline unit comprises at least one hardware accelerated convolutional neural network. In another embodiment, the visual pipeline unit is pre-trained using a training set of video and audio data suitable for a dedicated video conferencing space. In another embodiment, the dedicated video conferencing space is selected from the group consisting of a classroom, a workshop, a town hall, a newsroom, a boardroom, a courtroom, an interview studio, and a voting room.

[0012] In another embodiment, the predetermined rule set includes a first rule for evaluating possible block diagrams for each object in the video conference space based on a first plurality of parameters to determine an optimal block diagram. In yet another embodiment, the first plurality of parameters include: (i) whether the object is speaking; (ii) the length of speaking time; (iii) the direction of the object's gaze; (iv) the degree of visibility of the object in the block diagram; (v) the posture of the object; and (vi) which other objects are visible in the block diagram.

[0013] In another embodiment, the predetermined rule set further comprises a second rule for detecting a change in the video conference space based on a second plurality of parameters to trigger a transition of the block diagram. In another embodiment, the second plurality of parameters comprises: (i) the subject begins to speak; (ii) the subject moves; (iii) the subject stands up; (iv) the direction of the subject's gaze changes; (v) the subject shows a reaction; (vi) the subject shows a new item in the scene; (vii) the subject has been speaking for a predefined length of time; and (viii) there is a lack of meaningful reactions in other subjects within a predefined length of time.

[0014] In yet another embodiment, the predetermined rule set further includes a third rule for applying an appropriate shot type to each frame based on a third plurality of parameters consistent with television studio production principles. In another embodiment, the third plurality of parameters includes: (i) an overall shot for framing substantially all objects and a majority of the video conferencing space, thereby providing an overall context to the video conferencing space; (ii) a medium shot for framing a predetermined number of objects and focusing on the person who is speaking, thereby featuring an active conversation; and (iii) a close shot for framing one object that speaks for a predefined length of time, thereby featuring a presenter.

[0015] In another embodiment, the third plurality of parameters further includes: (i) an interest lens for framing objects of interest based on cues from a scene in the video conferencing space, the objects of interest including objects at the center of gaze of each object in the video conferencing space and items lifted by the objects; (ii) a listening lens for framing at least one object that is not speaking, thereby featuring the participation of the non-speaking object in the video conferencing space; and (iii) a presenter lens for framing an object that has been speaking for a predefined longer length of time than other objects, thereby featuring a presenter from different camera angles and compositions in the video conferencing space. According to various embodiments, the interest lens is suitable as a close lens, the listening lens is suitable as one of a close lens and a medium lens, and the presenter lens is suitable as one of a close lens and a medium lens.

[0016] In another embodiment, the predetermined rule set also includes a fourth rule for applying a virtual director's cut to the video conferencing space based on a fourth plurality of parameters, thereby customizing it as a dedicated television production scene. In yet another embodiment, the fourth plurality of parameters include: (i) a classroom production scene that begins with using a total shot to show the presenter and the audience, then transitions to framing the presenter in a presenter shot for a predefined length of time, and then switches between a listening shot showing the audience and a presenter shot showing the presenter; (ii) a workshop production scene that begins with using a total shot to show all participating objects, then transitions to using a mid-shot to show active objects, then using a mid-shot to show objects moving in the video conferencing space, and after a predefined length of time rotates back to show active objects in the video conferencing space; and (iii) a meeting A conference room production scene that begins by creating an overall shot of the entire video conferencing space with all visible objects, after a predefined length of time, transitions to framing a group of objects in a sub-location of the video conferencing space with a medium shot to focus on the active object, then frames the object speaking at the sub-location with a medium shot that best shows the front of the object's face, after another predefined length of time, switches to framing other objects in the video conferencing space with a listening shot that best shows the front of the object's face, and if no objects in the video conferencing space are speaking, rotates back to an overall shot featuring all objects.

[0017] According to various embodiments, the active objects include an object that is speaking, a whiteboard on which content is being drawn, and a presentation item held up by an object, and the sub-locations of the video conference space include a side of a table, a stage, a bench, a podium, and a row of chairs. In another embodiment, the conference room production scene is further adapted to represent a production scene, the production scene being selected from the group consisting of a news broadcast or podcast, a bilateral negotiation, a court proceeding, a panel discussion, and a voting convention.

[0018] In another embodiment, the predetermined rule set also includes a fifth rule for constructing clean shots of objects within the virtual meeting space. The fifth rule includes: not selecting any shots with partially visible objects, and aligning the eyes of the active object in the top third of the frame. In yet another embodiment, the fifth rule also includes: adding space filling in the direction of the object's gaze, and framing adjacent active objects together using the mid-shot. According to various embodiments, the adjacent active objects include objects, whiteboards, display monitors, lecterns, podiums, and presentation items.

[0019] In another embodiment, the plurality of smart cameras further comprises at least one peripheral smart camera placed in a separate video conferencing space. The predetermined set of rules in the master camera is further adapted to create an automatic television studio production for the combined virtual conference space.

[0020] In another embodiment, the autonomous video conferencing system further comprises at least one smart sensor located in the separate video conferencing space. The smart sensor is adapted to capture non-image signals in the separate video conferencing space and provide input to the virtual director unit of the main camera.

[0021] In yet another embodiment, the application programming interface of the master camera is adapted to receive input from the user computer to modify a predetermined set of rules for the virtual director unit of the master camera.

[0022] According to the present disclosure, in another embodiment, a method for creating an automatic TV studio production for a video conference space with the assistance of a virtual director is provided. The method includes: using multiple image sensors to capture video images in the video conference space; using multiple audio devices to capture audio signals in the video conference space; generating an overview video stream and a focus video stream, wherein the focus video stream includes a sub-video image that frames a detected object within the overview video stream; using machine learning to detect objects and gestures from the overview video stream and the audio signal, wherein the objects include people and non-human items, and the gestures include the position, orientation, gesture, and direction of the detected objects; implementing a virtual director, wherein the virtual director includes a predetermined set of rules that conform to the principles of TV studio production; applying the predetermined set of rules to the detected objects and gestures, thereby framing the objects of interest and updating the focus video stream in real time; and outputting the updated focus video stream to a user computer.

[0023] In yet another embodiment, the machine learning is performed on at least one hardware accelerated convolutional neural network. In another embodiment, the neural network is pre-trained using a training set of video and audio data suitable for a dedicated video conferencing space. In another embodiment, the dedicated video conferencing space is selected from the group consisting of a classroom, a workshop, a town hall, a newsroom, a boardroom, a courtroom, an interview studio, and a voting room.

[0024] According to various embodiments, the predetermined rule set includes: a first rule, wherein the first rule is used to evaluate possible block diagrams for each object in the video conferencing space based on a first plurality of parameters to determine the best block diagram; a second rule, wherein the second rule is used to detect changes in the video conferencing space based on a second plurality of parameters to trigger a conversion of the block diagram; a third rule, wherein the third rule is used to apply a suitable lens type to each block diagram based on a third plurality of parameters consistent with television studio production principles; a fourth rule, wherein the fourth rule is used to apply a virtual director's cut to the video conferencing space based on a fourth plurality of parameters to customize it into a dedicated television production scene; and a fifth rule, wherein the fifth rule is used to construct a clean shot for objects within the virtual conference space based on a fifth plurality of parameters.

[0025] In another embodiment, the shot type is selected from the group consisting of a general shot, a medium shot, a close shot, an interest shot, a listening shot, and a presenter shot. In yet another embodiment, the dedicated television production scene includes a classroom, a workshop, a conference room, a broadcast, a bilateral negotiation, a court proceeding, a panel discussion, and a voting conference.

[0026] In another embodiment, the method for creating an automated television studio production further comprises capturing non-image signals in the video conferencing space using a plurality of smart sensors. Each of the plurality of smart sensors comprises an application programming interface that interfaces with the virtual director to provide input to the virtual director. In various embodiments, each of the plurality of smart sensors is selected from the group consisting of a touch pad, a microphone, a smart phone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An autonomous video conferencing system according to one embodiment is described.

[0028] Figure 2 Depicted are an overview video stream and a focus video stream according to one embodiment.

[0029] Figure 3 Total shots, mid shots, and close shots are shown according to one of the rules in a predetermined rule set in one embodiment.

[0030] Figure 4 It is shown that in one embodiment, a clean shot is constructed according to one of the rules in a predetermined rule set.

[0031] Figure 5 Alignment for constructing a clean shot according to one of the rules in a predetermined rule set is shown in one embodiment.

[0032] Figure 6Two examples of small video conferencing spaces for specialized scenarios in which an autonomous video conferencing system is deployed are shown according to one embodiment.

[0033] Figure 7 Two examples of medium-sized video conferencing spaces for a dedicated scenario in which an autonomous video conferencing system is deployed are shown according to one embodiment.

[0034] Figure 8 Two examples of large video conferencing spaces for specialized scenarios in which an autonomous video conferencing system is deployed are shown according to one embodiment. DETAILED DESCRIPTION

[0035] The video conferencing system and method of the present disclosure are powered by multiple smart cameras and smart sensors using sub-symbolic and symbolic artificial intelligence. In various embodiments, the autonomous video conferencing system of the present disclosure includes multiple smart cameras optionally coupled to multiple smart sensors located in a video conferencing space. Each smart camera is equipped with: a visual pipeline supported by machine learning to detect objects and gestures and discern speaker-listener dynamics; and a virtual director suitable for applying a predetermined set of rules consistent with the principles of television studio production. Methods for creating automated television studio productions for various meeting spaces and specialized scenes using virtual directors are also provided.

[0036] Multiple smart cameras and sensors

[0037] Reference Figure 1 According to one embodiment, an autonomous video conferencing ("AVC") system includes a plurality of smart cameras (100, 101) and optionally one or more smart sensors (201). One of the plurality of smart cameras is adapted to be the primary camera and the remaining smart cameras of the plurality of smart cameras are adapted to be the peripheral cameras (101). Each of the plurality of smart cameras includes an image sensor (201, 301), an audio device (204, 304), a visual pipeline unit or visual pipeline (202, 302), a virtual director unit or virtual director (203, 303), a stream selector (205, 305) and an application programming interface ("API") (206, 306). The one or more smart sensors each include an API (401) that is connectable to the API of the smart camera. According to one embodiment, the API of the primary camera (100) is connected to the API of each peripheral camera (101) and the API of each smart sensor (102). The API of the primary camera is also adapted to provide a connection to a user computer (103).

[0038] In various embodiments, multiple smart cameras and one or more smart sensors in the AVC system are connected via Ethernet, other local area networks, or wireless networks. According to one embodiment, the main camera and peripheral cameras are located differently in the video conferencing space to provide effective coverage of the video conferencing space. In another embodiment, multiple smart sensors are activated and strategically located in the video conferencing space to capture non-image signals and provide input to the main camera of the AVC system. The smart sensors of the present disclosure include touch pads, microphones, smart phones, GPS trackers, echolocation sensors, thermometers, humidity sensors, and biometric sensors in various embodiments.

[0039] In alternative embodiments, one or more peripheral cameras and smart sensors of the AVC system are placed in a separate video conferencing space as a secondary space for video conferencing. These peripheral cameras and smart sensors are networked with the primary camera and are adapted to provide image and non-image inputs from the secondary space to the primary camera. Thus, the AVC systems of these alternative embodiments are also adapted to generate an automated television studio production for the combined video conferencing space based on inputs from all cameras and smart sensors in both spaces.

[0040] In various embodiments, the smart cameras of the AVC system are adapted to have different fields of view. For example, where the video conferencing space is small and the AVC system has a small number of multiple cameras, the smart camera may have a wide field of view, for example, about 150 degrees. On the other hand, where the video conferencing space is large and the AVC system has a large number of multiple cameras, the smart camera may have a narrower field of view, for example, about 90 degrees. The AVC system in another embodiment is equipped with smart cameras with various fields of view, thereby allowing optimal coverage of the video conferencing space. In another embodiment, the image sensor (201, 301) is adapted to be magnified to 10X, thereby achieving close-up images of objects at the far end of the video conferencing space. In an alternative embodiment, one or more smart cameras in the AVC system are adapted to capture content on or around objects (e.g., whiteboards, TV displays, posters, and presentation tables) that are non-human items within the video conferencing space. These cameras may be smaller and placed differently from other smart cameras in the AVC system, and may be mounted in the ceiling to provide effective coverage of the target content.

[0041] In another embodiment, the audio device (204, 304) in the smart camera is a microphone array adapted to pick up audio signals from different locations around the camera. Using the signals from the different microphones allows the smart camera to determine the audio direction ("DOA") and discern whether there is silence in a location or direction. This information in turn can be used by the visual pipeline (202, 302) and virtual director (203, 303) of the AVC system. In another alternative embodiment, a computing device with high computing power is connected to the AVC system through an Ethernet switch and is adapted to provide additional computing power to the AVC system. It has one or more high-performance CPUs and GPUs and runs parts of the visual pipeline for the main camera and any designated peripheral cameras.

[0042] Thus, the inclusion and configuration of multiple smart cameras and smart sensors in an AVC system allows for efficient and robust coverage of a variety of video conferencing spaces and scenes. By placing multiple smart cameras in a video conferencing space, the AVC system of the present disclosure creates a richer, natural, and engaging experience for all participants (including remote users), wherein the multiple smart cameras collaborate to frame objects in the conference from different camera angles and zoom levels.

[0043] Visual Pipeline; Overview and Focus Streams

[0044] Reference Figure 2 For each smart camera in the AVC system, there are two video streams, an overview stream and a focus stream. The overview stream sees the entire scene, which is composed of Figure 1 The visual pipeline (202, 302) shown consumes. The focus stream is a high-resolution stream that frames objects of interest depending on activity over time in the video conference space. This is the case when video settings are applied under the guidance of the virtual director (203, 303), thereby converting the focus stream into an updated focus stream, as discussed in detail below.

[0045] The visual pipeline of the present disclosure is adapted to process incoming overview streams and audio signals and detect objects and gestures with the aid of machine learning. In various embodiments, objects include people and non-human items, and gestures include position, orientation, gesture, and direction. In one embodiment, the visual pipeline includes one or more hardware-accelerated programmable convolutional neural networks that employ pre-trained weights to allow detection of certain attributes of objects in the field of view. For example, the visual pipeline detects where objects are in the field of view of the smart camera, their degree of visibility in the field of view, whether they are speaking, their facial expressions, their body position, and their head posture. The visual pipeline also tracks each object over time and determines: where the object was previously in the field of view, whether they are moving, and in what direction they are moving.

[0046] According to various embodiments, one advantage of a visual pipeline that utilizes sub-symbolic artificial intelligence to detect objects and their activities and interactions is that these convolutional neural networks are trained to not be biased on attributes such as gender, age, race, scene, light, and size. This enables the AVC system to create a more accurate and natural video stream production for the entire scene and all objects in the video conferencing space.

[0047] The vision pipeline of various embodiments is adapted to run on a GPU or other dedicated chipset with hardware accelerators for the relevant mathematical operations in its convolutional neural network. In an alternative embodiment, the vision pipeline runs on the available CPU capacity in the AVC system. In certain embodiments, the vision pipeline is further optimized to adapt to its hardware chipset by replacing the mathematical operations in its convolutional neural network architecture with equivalent mathematical operations supported by the chipset. Therefore, the dedicated hardware support of the vision pipeline allows it to perform detection of objects and gestures at high frequency and fast processing time, which in turn enables the AVC system to react quickly and on demand to changes in the field of view of its smart camera.

[0048] In some embodiments, the visual pipeline is pre-trained by running thousands of images and videos related to the scenes and detection targets of the video conferencing space. During training, the visual pipeline is evaluated using a loss function that measures the extent to which the visual pipeline performs a specific detection. Feedback from the loss function is then used to adjust the weights and parameters of the visual pipeline until it performs a specific detection at a predefined level of satisfaction. In one embodiment, the visual pipeline is further pre-trained and fine-tuned with a training set of video and audio data that is suitable for a dedicated video conferencing space for a dedicated scene. For example, the visual pipeline can be fine-tuned for a classroom, workshop, town hall, newsroom, boardroom, courtroom, interview studio, or voting room to support dedicated scenes such as speeches, interviews, news broadcasts or podcasts, court proceedings, workshops, bilateral negotiations, or voting conferences.

[0049] In various embodiments, the visual pipeline is also adapted to gather and process audio signals from microphones or other audio devices in the AVC system. It is able to distinguish speech, including whether an object is raising or lowering speech, depending on what is happening in the video conferencing space. In one embodiment, the visual pipeline is adapted to classify the topic of conversation based on the audio signal. Speech that does not belong to the object is classified as artificial sound and can be attributed to other sources such as speakers. The visual pipeline combines speech classification and characteristics with other information and knowledge it detects and collects from image data for related objects and their activities and interactions, thereby generating a comprehensive understanding of the video conferencing space and all detected objects. The visual pipeline makes this comprehensive corpus of understanding available to the virtual director, who in turn is responsible for selecting the best shots and creating an automated television studio production from the video conferencing space.

[0050] Virtual Director; Predetermined Rulesets

[0051] As described above, the virtual director (203, 303) of the AVC system is connected to the Figure 1 The visual pipeline (202, 302) and audio devices (204, 304) shown are adapted to block objects of interest according to a predetermined set of rules, thereby allowing real-time updating of the focus stream. The predetermined set of rules is developed and implemented for a virtual director in accordance with television studio production principles, as discussed in detail below. The conversion of the focus stream to the updated focus stream is performed by a stream selector (205, 305) under the direction of the virtual director in each smart camera.

[0052] Multiple smart cameras of the AVC system seamlessly collaborate to generate and update focus streams for various locations and target objects within the video conferencing space. The virtual director (203) of the main camera is connected to the virtual director (303) of each peripheral camera through their respective APIs (206, 306), such as Figure 1 As shown. The stream selector (205) of the master camera is connected to the stream selector (306) of each peripheral camera and is adapted to consume the focus streams from the master camera and all peripheral cameras. The stream selector (205) of the master camera is also responsible for selecting one of the updated focus streams from all cameras in the AVC system as the updated main focus stream under the guidance of the virtual director (203) of the master camera. The updated main focus stream is as follows Figure 1 The streaming output available to the user computer is shown. The virtual director (203) of the master camera is therefore the brains or command center of the AVC system and is responsible for creating an automated television studio production for the video conferencing space based on input from all cameras and any smart sensors deployed in the AVC system.

[0053] In various embodiments, the virtual director of the present disclosure is a software component that utilizes rule-based symbolic artificial intelligence to optimize decision making. The virtual director takes input from the visual pipeline and determines which frame diagram should be selected from which camera in the AVC system and ultimately streamed to the user. In certain embodiments, the virtual director does this by evaluating the possible frame diagrams of each object in the video conferencing space and its activities over time. For example, for each object captured by a specific smart camera in the AVC system, the virtual director evaluates different image crops in which the object is visible in order to find the best frame diagram for the object in the context of its activities and interactions. The virtual director then determines the video settings that are appropriate to convert the focus stream of a specific camera into the selected best frame diagram. The stream selector then performs conversions and updates of the focus stream using these video settings under the guidance of the virtual director.

[0054] In some embodiments, the virtual director of the primary camera additionally obtains input from one or more smart sensors deployed in the AVC system through their respective APIs. Input from smart sensors typically includes non-image signals or cues about the video conferencing space and all objects therein, such as the location, movement, and physiological or biological attributes of the objects.

[0055] The API of the AVC system is adapted to transmit messages between the various components over an internal network bus. These messages include information about the status of each smart camera, such as whether it is connected, what type of software it is running, current health status, etc. The API also transmits what the camera detects from the image data, such as where objects are detected in the image, where they are placed in the meeting space, and other information detected by the visual pipeline. The API additionally transmits video settings applied to the focus stream, such as image attributes, color, brightness, etc. In addition, the API is adapted to transmit virtual director parameters, which allows the AVC system to automatically set and adjust virtual director rule sets and related parameters for all of its component cameras, as well as allowing users of the AVC system to personalize the virtual director experience by modifying certain parameters and rules for their predetermined rule sets.

[0056] As described above, the virtual director of the present disclosure employs rule-based decision making using symbolic artificial intelligence. Predetermined rule sets are developed in various embodiments that conform to the principles of television studio production. This allows the AVC system to produce an automated television studio production of a conference experience that is similar to a television production directed by a real-world professional. In a certain embodiment, the predetermined rule set in each camera of the AVC system can be applied differently to its location and the target object or content within the video conference space. In one embodiment, as Figure 1As shown, the predetermined rule set applied by the virtual director (203) of the master camera includes more rules than the predetermined rule set applied by the virtual director (303) of the peripheral camera. In an alternative embodiment, the user is provided with an option to modify or time certain rules and parameters of the predetermined rule set in the AVC system through the API of the master camera, which is connected to the virtual director unit (203) of the master camera, such as Figure 1 shown.

[0057] According to one embodiment, the predetermined rule set includes a first rule for evaluating possible block diagrams for each object in the video conference space based on a first plurality of parameters. The first rule determines the best block diagram for each object. In one embodiment, the first plurality of parameters include (i) whether the object is speaking; (ii) the length of time the object is speaking; (iii) the direction the object is looking; (iv) the degree of visibility of the object in the block diagram; (v) the posture of the object; and (vi) which other objects are visible in the block diagram.

[0058] The predetermined rule set in another embodiment includes a second rule for detecting a change in the video conference space based on a second plurality of parameters. The second rule triggers a transition of the block diagram. In one embodiment, the second plurality of parameters includes: (i) the subject begins to speak; (ii) the subject moves; (iii) the subject stands up; (iv) the direction of the subject's gaze changes; (v) the subject shows a reaction; (vi) the subject shows a new item in the scene; (vii) the subject has been speaking for a predefined length of time; and (viii) there is a lack of meaningful reactions in other subjects within a predetermined length of time.

[0059] In another embodiment, the predetermined rule set includes a third rule for applying an appropriate lens type to each frame based on a third plurality of parameters. In one embodiment, the third plurality of parameters includes: (i) an overall lens for framing substantially all objects and a majority of the video conferencing space, thereby providing an overall context to the video conferencing space, see Figure 3 , top frame; (ii) a middle shot, which is used to frame a predetermined number of objects and focus on the person who is speaking, thereby featuring an active conversation, see Figure 3 , middle frame; and (iii) a close-up shot, which is used to frame an object speaking for a predefined length of time, thereby featuring the presenter, see Figure 3 , bottom box diagram.

[0060] The third plurality of parameters in another embodiment also includes: (i) an interest lens for framing objects of interest based on cues from a scene in the video conferencing space, the objects of interest including an object at the center of gaze of each object in the video conferencing space, and items lifted by the objects; (ii) a listening lens for framing at least one non-speaking object, thereby characterizing the participation of the non-speaking object in the video conferencing space; and (iii) a presenter lens for framing the object that has been speaking for the longest length of time compared to other objects, thereby characterizing the presenter from different camera angles and compositions in the video conferencing space. In various embodiments, the interest lens is adapted to be a close lens, the listening lens is adapted to be a close lens or a medium lens, and the presenter lens is adapted to be a close lens or a medium lens.

[0061] In another embodiment, the predetermined rule set includes a fourth rule for applying the virtual director's cut to the video conference space based on a fourth plurality of parameters. The fourth rule customizes the video settings for a dedicated television production scene. In one embodiment, the fourth plurality of parameters includes a classroom production scene, a workshop production scene, and a conference room production scene. In some embodiments, the conference room production scene is also suitable for news broadcasts or podcasts, bilateral negotiations, court proceedings, panel discussions, and voting conferences.

[0062] In another embodiment, the predetermined rule set includes a fifth rule for constructing clean shots for objects within the virtual meeting space. In one embodiment, the fifth rule includes not selecting any shot with partially visible objects. Figure 4 , for example, the left frame is a clean shot, while the right frame is not a clean shot and is not selected in the AVC system under the fifth rule. In another embodiment, the fifth rule includes aligning the eyes of the active object to the top third of the frame. Figure 5 , for example, each image (top, middle, and bottom) is optimally aligned with the eyes of the active subject in the top third of the block diagram. These shots are clean shots constructed and selected in the AVC system under the fifth rule.

[0063] In yet another embodiment, the fifth rule includes adding space filling in the direction of the object's gaze and framing adjacent active objects together using a mid-shot. In various embodiments, adjacent active objects include objects, whiteboards, display monitors, lecterns, podiums, and presentation items.

[0064] Dedicated scenes

[0065] In various embodiments, the AVC system of the present disclosure is applicable to various dedicated video conferencing spaces for dedicated video conferencing scenarios. According to certain embodiments, the visual pipeline unit of the smart camera in the AVC system is pre-trained using a training set of video and audio data suitable for dedicated video conferencing spaces, including classrooms, workshops, town halls, news rooms, boardrooms, courtrooms, interview studios, and voting rooms.

[0066] As described above, among the predetermined rule sets implemented and applied by the virtual director unit of the AVC system is a fourth rule for making a virtual director's cut based on a fourth plurality of parameters. The fourth rule for the virtual director's cut is designed to customize the video conferencing space for a dedicated television production scene. The fourth plurality of parameters in one embodiment includes a classroom production scene, a workshop production scene, and a conference room production scene.

[0067] In one embodiment, the classroom scene begins using an overall shot showing the presenter and audience, then transitions to framing the presenter in a presenter shot for a predefined length of time, and then switches between a listening shot showing the audience and a presenter shot showing the presenter.

[0068] In one embodiment, the workshop production scene begins with a general shot showing all participating objects, then switches to a mid-shot to show active objects, then switches to a mid-shot to show objects moving in the video conferencing space, and finally rotates back after a predefined length of time to show active objects within the video conferencing space.

[0069] In one embodiment, the conference room production scene begins by creating an overall shot of the entire video conferencing space with all visible objects, transitioning after a predefined length of time to framing a set of objects in a sub-location of the video conferencing space with a medium shot to focus on the active object, then switching to framing the object speaking at the sub-location using a medium shot that best shows the front of the object's face, then after another predefined length of time, switching to framing other objects in the video conferencing space using a listening shot that best shows the front of the object's face, and finally rotating back to an overall shot featuring all objects if no objects are speaking in the video conferencing space. In various embodiments, the active object represents an object that is speaking, a whiteboard on which content is being drawn, or a presentation item held up by an object. In various embodiments, the sub-location of the video conferencing space represents a side of a table, a stage, a bench, a podium, or a row of chairs.

[0070] The conference room production scenes in certain embodiments are also suitable for representing news broadcasts or podcasts, interviews, board meetings, bilateral negotiations, court proceedings, panel discussions, and voting conferences.

[0071] For various dedicated scenarios, the main camera and peripheral cameras are strategically and differently placed within the video conferencing space to provide effective coverage of the space. One or more optional smart sensors are additionally distributed within the video conferencing space, connected to the main camera, and provide input to the main camera's virtual director unit. Figure 6-8 An example of an AVC system of the present disclosure deployed for a dedicated scenario is shown in FIG. In these figures, each dot represents a smart camera or smart sensor of the AVC system, and each circle represents an object, including a person or non-person item, such as a chair or a presentation subject. The small rectangles along the sides of each of these figures represent TVs, whiteboards, posters, or projection displays (601, 602, 701, 702, 703, 704, 801, 802). The square or rectangle in the middle of the figure represents a table.

[0072] Reference Figure 6 , both the top and bottom depict a small video conferencing space in which the AVC system is deployed. The top configuration represents a small meeting and group discussion scenario in one embodiment. In one embodiment, the bottom configuration represents a news broadcast or podcast and interview scenario.

[0073] Reference Figure 7 , both the top and bottom depict a medium video conferencing space in which an AVC system is deployed. In one embodiment, the top configuration represents a board meeting scenario. In one embodiment, the bottom configuration represents a bilateral negotiation.

[0074] Reference Figure 8 , both the top and bottom depict a large video conferencing space in which the AVC system is deployed. In one embodiment, the top configuration represents a workshop scene. In one embodiment, the bottom configuration represents a classroom scene.

[0075] The description of the various embodiments including the figures and examples is intended to illustrate rather than to limit the invention and its various embodiments.

Claims

1. An autonomous video conferencing system, comprising a plurality of smart cameras, each of the plurality of smart cameras comprising: an image sensor adapted to capture video images in a video conferencing space and output an overview video stream and a focus video stream, wherein the focus video stream includes sub-video images framing detected objects within the overview video stream; an audio device adapted to capture audio signals in the video conferencing space; a vision pipeline unit adapted to receive the overview video stream and the audio signal and detect objects and gestures by means of machine learning, wherein the objects include humans and non-human items and the gestures include the position, orientation, gesture and direction of the detected objects; a virtual director unit connected to the visual pipeline unit and the audio device and adapted to frame objects of interest according to a predetermined set of rules, thereby allowing real-time updating of the focused video stream, wherein the predetermined set of rules is adapted to create an automated television studio production from the video conferencing space; a stream selector connected to the virtual director unit and adapted to convert the focused video stream into an updated focused video stream; and an application programming interface connected to the virtual director unit and adapted to provide at least one connection to another smart camera of the plurality of smart cameras and one connection to a user computer, wherein one of the plurality of smart cameras is adapted to be a main camera, and each of the remaining smart cameras of the plurality of smart cameras is adapted to be a peripheral camera, wherein an application programming interface of each peripheral camera is connected to an application programming interface of the main camera, wherein a stream selector of each peripheral camera is further adapted to transmit its updated focus stream to a stream selector of the main camera, and wherein the stream selector of the main camera is further adapted to select one of the updated focus streams from the plurality of smart cameras as an updated main focus stream, and output the updated main focus stream to a user computer.

2. The system according to claim 1, further comprising a plurality of smart sensors, the plurality of smart sensors being placed in the video conferencing space and being adapted to capture non-image signals in the video conferencing space, each of the plurality of smart sensors comprising an application programming interface, wherein The application programming interface is adapted to connect to an application programming interface of the primary camera so as to provide input to a virtual director unit of the primary camera.

3. The system according to claim 2, wherein: Each of the plurality of smart sensors is selected from the group consisting of a touch pad, a microphone, a smart phone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor.

4. The system according to claim 1, wherein: The audio device comprises a microphone array adapted to provide directionality of audio using captured audio signals.

5. The system according to claim 1, wherein: The visual pipeline unit includes at least one hardware-accelerated convolutional neural network.

6. The system according to claim 5, wherein: The visual pipeline unit is pre-trained using a training set of video and audio data suitable for a dedicated video conferencing space.

7. The system according to claim 6, wherein: The dedicated video conferencing space is selected from the group consisting of a classroom, a workshop, a town hall, a news room, a boardroom, a courtroom, an interview studio, and a voting room.

8. The system according to claim 1, wherein: The predetermined rule set includes a first rule for evaluating possible block diagrams for each object in the video conferencing space based on a first plurality of parameters to determine an optimal block diagram.

9. The system according to claim 8, wherein: The first plurality of parameters include: (i) whether the subject is speaking; (ii) the length of time the subject is speaking; (iii) the direction the subject is looking; (iv) the degree of visibility of the subject in the frame diagram; (v) the posture of the subject; and (vi) which other objects are visible in the frame diagram.

10. The system according to claim 8, wherein: The predetermined rule set also includes a second rule for detecting a change in the video conferencing space based on a second plurality of parameters to trigger a transition of the block diagram.

11. The system according to claim 10, wherein: The second plurality of parameters includes: (i) the subject begins to speak; (ii) the subject moves; (iii) the subject stands up; (iv) the direction of the subject's gaze changes; (v) the subject shows a reaction; (vi) the subject shows a new item in the scene; (vii) the subject has been speaking for a predefined length of time; and (viii) there is a lack of meaningful reaction from other subjects within a predefined length of time.

12. The system according to claim 10, wherein: The predetermined rule set also includes a third rule for applying an appropriate shot type to each block diagram based on a third plurality of parameters consistent with television studio production principles.

13. The system according to claim 12, wherein: The third plurality of parameters includes: (i) a total shot that frames substantially all objects and a majority of the video conferencing space, thereby providing an overall context to the video conferencing space; (ii) a medium shot that frames a predetermined number of objects and focuses on the person speaking, thereby featuring an active conversation; and (iii) a close shot that frames one object speaking for a predefined length of time, thereby featuring a presenter.

14. The system according to claim 13, wherein: The third plurality of parameters also includes: (i) an interest lens for framing objects of interest based on cues from the scene in the video conferencing space, the objects of interest including objects at the center of gaze of each object in the video conferencing space and items lifted by the objects; (ii) a listening lens for framing at least one object that is not speaking, thereby characterizing the participation of the non-speaking object in the video conferencing space; and (iii) a presenter lens for framing the object that has been speaking for the longest length of time compared to other objects, thereby characterizing the presenter from different camera angles and compositions in the video conferencing space, wherein the interest lens is suitable as a close lens, the listening lens is suitable as one of a close lens and a medium lens, and the presenter lens is suitable as one of a close lens and a medium lens.

15. The system of claim 12, wherein: The predetermined rule set also includes a fourth rule for applying a virtual director's cut to the video conferencing space based on a fourth plurality of parameters to customize it as a dedicated television production scene.

16. The system of claim 15, wherein: The fourth plurality of parameters comprises: (i) a classroom production scene that begins with a total shot showing the presenter and the audience, then switches to framing the presenter in a presenter shot for a predefined length of time, then switches between a listening shot showing the audience and a presenter shot showing the presenter; (ii) a workshop production scene that begins with a total shot showing all participating objects, then switches to using a mid-shot to show active objects, then using a mid-shot to show objects moving in the video conferencing space, and after a predefined length of time rotates back to show active objects in the video conferencing space; and (iii) a conference room production scene that begins an overall shot to create an understanding of the entire videoconferencing space with all visible objects, after a predefined length of time, transition to framing a group of objects in a sub-location of the videoconferencing space with a medium shot to focus on the active object, then framing an object speaking at the sub-location with a medium shot that optimally shows the frontal face of the object, after another predefined length of time, switching to framing other objects in the videoconferencing space with a listening shot that optimally shows the frontal face of the object, and if no object in the videoconferencing space is speaking, rotating back to an overall shot featuring all objects, wherein the active objects include a subject who is speaking, a whiteboard on which content is being drawn, and a presentation item held up by the subject, and wherein the sub-locations of the video conferencing space include one side of a table, a stage, a bench, a podium, and a row of chairs.

17. The system of claim 16, wherein: The conference room production scene is further adapted to represent a production scene selected from the group consisting of a news broadcast or podcast, a bilateral negotiation, a court proceeding, a panel discussion, and a voting conference.

18. The system of claim 15, wherein: The predetermined rule set also includes a fifth rule for constructing clean shots of objects within the virtual meeting space, the fifth rule including: not selecting any shot with partially visible objects; and aligning the eyes of active objects in the top third of the frame.

19. The system of claim 18, wherein: The fifth rule also includes: adding space filling in the direction of the object's gaze; and framing adjacent active objects together using the mid-lens, wherein the adjacent active objects include objects, whiteboards, display monitors, lecterns, podiums, and presentation items.

20. The system of claim 1, wherein: The plurality of smart cameras further comprises at least one peripheral smart camera placed in a separate video conferencing space, wherein the predetermined rule set in the master camera is further adapted to create an automated television studio production for the combined virtual conferencing space.

21. The system of claim 20, further comprising at least one smart sensor located in the separate video conferencing space, wherein: The smart sensor is adapted to capture non-image signals in the separate video conferencing space and provide input to the virtual director unit of the main camera.

22. The system of claim 1, wherein: The application programming interface of the master camera is adapted to receive input from the user computer to modify a predetermined set of rules for the virtual director unit of the master camera.

23. A method for creating an automated television studio production for a video conferencing space using a virtual director's assistance, comprising: capturing video images in the video conferencing space using a plurality of image sensors; capturing audio signals in the video conferencing space using a plurality of audio devices; Generate an overview video stream and a focus video stream, wherein the focus video stream includes a sub-video image that frames a detected object within the overview video stream; detect objects and gestures from the overview stream and the audio signal by means of machine learning, wherein the objects include humans and non-human items, and the gestures include the position, orientation, gesture, and direction of the detected objects; implement a virtual director, wherein the virtual director includes a predetermined set of rules that conform to television studio production principles; apply the predetermined set of rules to the detected objects and gestures to frame the object of interest and update the focus video stream in real time; and output the updated focus video stream to a user computer.

24. The method according to claim 23, wherein: The machine learning is performed on at least one hardware accelerated convolutional neural network.

25. The method according to claim 24, wherein: The neural network is pre-trained using a training set of video and audio data suitable for a dedicated video conferencing space.

26. The method according to claim 25, wherein: The dedicated video conferencing space is selected from the group consisting of a classroom, a workshop, a town hall, a news room, a boardroom, a courtroom, an interview studio, and a voting room.

27. The method according to claim 23, wherein: The predetermined rule set includes a first rule for evaluating possible block diagrams for each object in the video conferencing space based on a first plurality of parameters to determine an optimal block diagram.

28. The method according to claim 27, wherein: The predetermined rule set also includes a second rule for detecting a change in the video conferencing space based on a second plurality of parameters to trigger a transition of the block diagram.

29. The method according to claim 28, wherein: The predetermined rule set also includes a third rule for applying an appropriate shot type to each block diagram based on a third plurality of parameters consistent with television studio production principles.

30. The method of claim 29, wherein: The shot type is selected from the group consisting of a general shot, a medium shot, a close shot, an interest shot, a listening shot, and a presenter shot.

31. The method according to claim 30, wherein: The predetermined rule set also includes a fourth rule for applying a virtual director's cut to the video conferencing space based on a fourth plurality of parameters to customize it as a dedicated television production scene.

32. The method according to claim 31, wherein: The dedicated television production scenarios include classrooms, workshops, meeting rooms, broadcasts, bilateral negotiations, court proceedings, panel discussions and voting conferences.

33. The method of claim 32, wherein: The predetermined rule set also includes a fifth rule for constructing a clean shot for an object within the virtual meeting space based on a fifth plurality of parameters.

34. The method of claim 23 further comprises capturing non-image signals in the video conferencing space using a plurality of smart sensors, each of the plurality of smart sensors comprising an application programming interface connected to the virtual director, thereby providing input to the virtual director.

35. The method of claim 34, wherein: Each of the plurality of smart sensors is selected from the group consisting of a touch pad, a microphone, a smart phone, a GPS tracker, an echolocation sensor, a thermometer, a humidity sensor, and a biometric sensor.