Multi-modal data-stream-based artificial intelligence interventions in virtual environment system and method

A multimodal AI system enhances video conferencing by detecting emotional signals and adjusting the virtual environment to improve engagement and interaction, addressing the monotony of traditional meetings and boosting productivity.

JP2024160978APending Publication Date: 2024-11-15TMRW FOUNDATION IP SARL
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
JP2024073841
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-02
Filing Date
2024-04-30
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Traditional video conferencing technologies often result in uninteresting and unproductive meetings due to a lack of emotional engagement and interaction, leading to decreased productivity and potential misunderstandings among participants.

Method used

A multimodal data stream-based AI system that processes multiple inputs such as camera and audio feeds to detect emotional signals from participants, adjusting the virtual environment's mood and dynamics in real-time through interventions like lighting, sound effects, and camera adjustments to enhance engagement and interaction.

Benefits of technology

The system creates a more dynamic and interactive virtual meeting experience by replicating natural human behavior, increasing user motivation, attention span, and productivity by analyzing emotional cues and adapting the environment accordingly.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160978000001_ABST
    Figure 2024160978000001_ABST
Patent Text Reader

Abstract

To provide systems and methods for enhancing a video conference technology using artificial intelligence (AI).SOLUTION: Systems and methods for an improved video conferencing system in a virtual environment can use artificial intelligence, large language models, and machine learning to detect empathy and emotion within a group of participants in the virtual conference. Various inputs, such as audio / visual streams, other sensor data, and training sets may be used to detect various emotional scenarios, such as boredom, happiness, and sadness. Embodiments may take an intervention based on a desired behavioral change in the participants, such as encouraging more participation or waking people up. The intervention may be a change in a camera perspective or an environmental change to the virtual environment. The intervention may also depend on the contextual scenario, e.g., a serious business meeting or a gathering of friends.SELECTED DRAWING: Figure 9
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to creating content in a computing environment, and more particularly to systems and methods for enhancing video conferencing technology using artificial intelligence (AI). AI may be used to develop a more engaging experience by creating a more interactive viewing experience beyond the typical speaker grid or speaker focus. [Background technology]

[0002] Video conferencing has become an essential tool for communication and collaboration in today's world. However, traditional forms of video conferencing often result in unengaging and unproductive meetings, leading to a lack of interest and attention from participants. Boring video conferencing can lead to reduced collaboration and communication among team members as well as reduced productivity and efficiency in achieving the meeting's objectives. Therefore, there is a need to improve video conferencing to make it more interesting and engaging for participants. This can be achieved through developing innovative technologies and features that enhance the user experience and encourage active participation, resulting in more productive and successful meetings.

[0003] As remote work and virtual collaboration become increasingly common, the limitations of traditional video conferencing formats have become more apparent: Participants often struggle to maintain focus and engagement during lengthy or unstructured meetings, leading to lost productivity and missed opportunities for collaboration and problem solving.

[0004] Moreover, the lack of face-to-face interaction and non-verbal emotional cues can make it difficult for participants to build rapport and establish trust, leading to potential misunderstandings and conflicts. This highlights the need for new approaches to video conferencing that foster more dynamic and interactive communication between participants.

[0005] In response to these challenges, there is an increasing demand for innovative video conferencing technologies that can improve the overall meeting experience. Some of these technologies include virtual backgrounds, augmented reality overlays, interactive whiteboards, and real-time collaboration tools. These features can help engage participants, facilitate more meaningful conversations and collaboration, and ultimately enhance productivity and decision-making.

[0006] As such, there is a clear need for continued investment and development in video conferencing technology to make it more engaging, interactive, and productive so that we can improve the virtual meeting experience and foster better communication, collaboration, and problem-solving among participants in remote or hybrid work environments. Summary of the Invention [Problem to be solved by the invention]

[0007] Frequent online meetings can become boring and tedious, reducing motivation to participate in such meetings. This is because humans are emotional beings by nature and tend to enjoy variety, and there are many types of stimuli that can achieve this. However, current technologies that enable communication in virtual environments are primarily focused on providing sufficient technical capabilities to enable a smooth user experience, such as by enabling adequate FPS, sound quality, or graphics quality. Therefore, there is a need to develop technological alternatives that can stimulate a wide range of human emotions to improve users' energy, attention span, motivation, and even productivity when interacting with each other in virtual environments. Virtual environments can include indoor environments, such as an office or a restaurant, and outdoor environments, such as a park or a farm.

[0008] Prior art solutions offer a few basic building blocks that do not provide a cohesive solution to creating engaging video or virtual meetings. Video conferencing is well known in the art, with applications such as Zoom® and Teams®. However, virtual meetings may include traditional video conferencing and also more elaborate virtual environments emulating real-world scenarios such as restaurants or conference rooms. Some embodiments of the present disclosure provide improved auto-response AI logic based on dynamic information adapted to affect the mood of an event by processing inputs into desired outputs. Embodiments may include multi-modal data stream based AI camera systems and methods that process multiple inputs into outputs. Inputs may be user audio or video feeds provided by corresponding cameras and microphones as the user participates in a virtual event. Further inputs may be captured by one or more virtual broadcast cameras in the virtual environment. Each user has a user graphical representation, or avatar, in the form of their own camera feed that is presented in the virtual environment. The background of the camera feed with the user can be removed so that the avatar is only a cut-out image of the user that is transmitted in real time to the virtual environment. The system includes a server storing the virtual environment and configured to process and distribute the received camera feeds from multiple participants. The system may use a WebRTC architecture using schemes such as P2P, client / server, or a combination thereof, as discussed in U.S. Patent No. 11,218,522.

[0009] Virtual environments provide an opportunity for users to meet and interact remotely. When interacting in such a virtual environment, users may need to actively adjust their camera view to be able to follow the action in the virtual environment. For example, in a round table with many users, when passing the turn from one speaker to another, users may need to make one or more movements to adjust their camera view to be able to see the new speaker. Patent application 17 / 060,516 by Yerli (incorporated herein by reference in its entirety) proposes a virtual broadcast camera that can be used to capture the action in the virtual environment that can be directly viewed by the user. However, for that case, a client device is needed to manage the virtual broadcast camera, and manual human effort is also required.

[0010] Apart from capturing live action that is provided to users, footage taken by virtual cameras, such as virtual broadcast cameras, can further be used to create replays of real or virtual events. A typical method requires the video stream to be sent to a storage device in a server, whereby a controller reviews and manually selects and / or edits shots that may be of interest to the viewer, involving a lot of effort. Finally, a typical virtual event follows a certain predefined sequence, which may make the meeting boring and monotonous, all of which is exacerbated by the remote nature of such events, leading to fatigue and boredom.

[0011] According to certain aspects of the present disclosure, a system and method are provided for a multimodal data stream based AI camera system that processes multiple inputs into outputs. The inputs can be camera and audio feeds shared by users with a virtual environment. The system comprises a server storing the virtual environment and configured to process and distribute received camera feeds from multiple participants. The system allows configuration of various possible outputs depending on the received inputs. The system further uses presets or context scenarios to allow programmatic selection of outputs and / or modifying the outputs. The outputs can be presented in the virtual environment on one or more surfaces thereof, for example, on a virtual screen of the virtual environment on which the event is broadcast, and can be configured to create a more engaging virtual environment. The camera outputs can take one or more forms of a live, unmodified camera feed, a feed of real-time camera adjustments, or a replay of the scene, for example to create a "best of" that includes the most relevant scenes of the event.

[0012] For example, if the system detects emotional signals that a participant is expressing enthusiasm and excitement, the system may adjust the mood of the event to match this emotion. This may be done by adjusting lighting and sound effects, changing the speaker's tone and body language, and adjusting the pace of the conversation. Similarly, if the system detects that a participant is feeling bored or disinterested, the system may adjust the mood to inject energy and excitement into the conversation.

[0013] A user's emotional signals may be based on several factors, such as body language, facial expressions, tone of voice, and word choice. Below are examples of emotional signals:

[0014] Gestures: Crossing the arms or legs may indicate defensiveness or discomfort. Leaning forward can indicate interest or concern. Fidgeting or tapping can indicate nervousness or impatience. Looking down or avoiding eye contact can indicate disinterest or discomfort. An open, relaxed posture can indicate confidence and ease.

[0015] Facial Expressions: Smiling can indicate joy or friendliness. Frowning or brow furrowing can indicate sadness or anger. Raising eyebrows can indicate surprise or interest. A pursed mouth or pursed lips can indicate anger or tension. Squinting or narrowing of the eyes can indicate doubt or disbelief.

[0016] Tone of voice: A high-pitched or quavering voice may indicate nervousness or fear. A monotone voice may indicate boredom or disinterest. A loud or raised voice may indicate anger or frustration. A soft, slow voice can indicate sadness or anxiety. A fast, energetic voice may indicate excitement or enthusiasm.

[0017] Word Choice: Positive words such as "wow," "great," and "great" may indicate delight or enthusiasm. Negative words such as "horrible," "disgusting," and "disappointing" may indicate anger or disappointment. Neutral words or phrases such as "interested," "not so-so," or "don't care" may indicate disinterest or indifference. Words or phrases that reflect personal experience, such as "I feel" or "in my opinion," may indicate a personal emotional response.

[0018] Datasets can be trained to teach neural networks to recognize these emotional signals.

[0019] The use of machine learning to detect emotions may also enhance the naturalness and spontaneity of a virtual event. In a face-to-face conversation, participants naturally adjust their tone, body language, and speech patterns based on the emotions of other participants. By using machine learning to detect emotions, the system may replicate this natural behavior in a virtual event. The system may analyze the emotions of participants and adjust the conversation to reflect these emotions to create a more natural and spontaneous conversation. Embodiments may use additional inputs such as whether people are talking, whether people are using hand gestures and the type of those gestures, body language, or other inputs received from a keyboard or other sensors, such as touch, pressure, temperature, humidity, light, heart rate, etc. Additional information that embodiments may process is how a user is interacting with or moving in a virtual environment.

[0020] Further inputs may include so-called "dead" information, such as background audio and video information that is cut out by the cutout process. There may be security or privacy concerns, which may be remedied by specific opt-in requirements or other legal measures. Noise cancellation processes may be used to extract or enhance foreground or key audio and classify and segment other sounds.

[0021] Each of these inputs can be analyzed, modeled, and processed in different ways and at different intervals to help select interventions that are meant to activate behavior in the virtual environment of a virtual meeting. For example, if someone is hearing a dog barking in their background, the system's extraction algorithms will recognize this and have a short list of criteria. The system may then add a video / animated 3D model of the dog in the scene outside the window. Or if someone turns on the kettle, the kettle in the room may start steaming. Or if it's prayer time, a golden light may light up in the sky. Cause and effect do not have to be direct or verbatim. They can be cumulative or combinatorial. For example, if two or three users simultaneously sound a team notification, the scale of a 3D object representing that sound may change. Or if the vacuuming noise is getting louder, a robotic vacuum cleaner may start moving closer to the user while making that sound. Objects in the room may represent the cumulative background noise level in the room in novel and artistic ways, such as water ripples on a sphere, fireworks outside, or subtle shaking of fixtures / furniture.

[0022] There can be a similar process for the background information of the video. If someone has a cup on their desk, this can be simulated into a real scene. If someone has green curtains behind them, one of the accent details of the virtual environment can change. If it is sunny outside the window, the weather outside the 3D room can change. Again, this can be combinatorial. So if location A is sunny and location B is cloudy, there can be sunshine shining through the clouds, or a rainbow. Or there can be a shared bookshelf or art wall in the virtual environment, where the reading material and background art tastes of all the people there are collected. Furthermore, if someone has a beach background, the background of the virtual environment can be a beach scene. If someone has a Star Trek poster, the background can be outer space, and so on.

[0023] More generally, if someone's lighting is varying between two thresholds, the virtual environment can adjust to this behavior. If someone has a piece of art in their background, this can be identified and applied to their personal virtual environment. Or if someone is wearing a branded product or using branded technology, this can be de-branded / blurred. If someone has a book on their bookshelf, this can be imported and linked for others to see, instead of pasting a URL into a chat. If someone holds an object up to the camera, this recognition process can help remove the friction of digitizing analog information using AI segmentation from OCR, for example.

[0024] Embodiments can also recreate 3D space using one or more 2D images, e.g., using neural luminance fields. For example, someone can point their phone at a chessboard, and a matching 3D playable chessboard may appear on a table in the virtual environment.

[0025] In virtual events, such as virtual conferences or webinars, it may be difficult to replicate the natural flow and mood of face-to-face conversations. However, by using machine learning to detect emotions, the system can simulate realistic, natural conversations in a painless and easy manner. Overall, the use of machine learning to detect emotions can have significant advantages in increasing the dynamism, naturalness, spontaneity, excitement, and energy of virtual events. The system can analyze participants' emotions in real time and adjust the tone and mood of the event accordingly to create a more engaging and dynamic experience. By replicating the natural behavior of face-to-face conversations, the system can also enhance the naturalness and spontaneity of the virtual event.

[0026] The embodiments include a method and a system for running an interactive virtual conference platform that stores multiple virtual environments in one or more virtual environment systems. The platform may also store multiple context scenarios, such as a business meeting or a party. Emotional signals may also be stored by the platform. The platform may receive multiple requests to join the virtual conference. Participants may connect to the virtual conference by connecting together sessions that include video and audio streams. The platform may include a server for analyzing the multiple video or audio streams using one or more neural networks. The analysis may automatically detect a context scenario of the multiple context scenarios or one or more emotional signals from the input data. Based on the analyzed input data, the system and method may select an intervention from an intervention database based on the analyzed input data, the detected context scenario, or the detected emotional signals. The platform may then retrieve the intervention from the intervention database. After retrieving the intervention, the platform may intervene in the virtual conference based on the intervention retrieved from the intervention database. The intervention may include at least one change to the output audio signal or the output video signal.

[0027] In some embodiments, the input data further includes one or more of typing speed, typing volume (cancellation), hand gestures, amount of speech time, facial (micro) expressions, mouse / swipe speed, geographic location, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, speech toxicity, device, or speech rhythm.

[0028] In some embodiments, the platform further comprises a configuration application that allows for setting certain options of the platform. For example, the configuration application may be configured to allow for the selection of an intervention independent of the context scenario in which the participants interact with each other. In another example, the configuration application may be configured to allow for the selection of an intervention based on input data related to the context scenario. In this example, input data in a particular context scenario may result in a different intervention depending on the context scenario. For example, if a participant raises their hand in a survey reality show type context scenario, the platform may select an intervention to put a spotlight on the participant and play a particular music, but if a participant raises their hand in a business context scenario, the platform may select an intervention to simply redirect the virtual camera angle to the participant and lower the microphones of all other participants without playing music or putting a spotlight on the participant.

[0029] The platform may also receive inputs including one or more datasets corresponding to the multiple context scenarios to one or more neural networks, which may include one or more of a convolutional neural network (CNN) and a recurrent neural network (RNN).

[0030] The system may be configured to receive feedback about the intervention and apply the feedback to one or more neural networks to train the one or more neural networks to apply to future interventions. The feedback may include a ranking from at least one user or a physical reaction from one or more users detected via a video or audio stream.

[0031] The intervention may include changes to the virtual camera angle, shot size, or virtual camera motion. The intervention may also include changing the brightness of the output video signal, changing the tone of the output audio signal, or changing the color hue of the output video signal.

[0032] Intervention may also include summarizing an event taking place in the virtual environment by selecting the most interesting scenes of the event and creating a summary thereof. In this embodiment, recordings may be made through one or more virtual broadcast cameras at the event capturing a wide range of data. The recordings may be presented on one or more surfaces of the virtual environment, such as on a screen of the virtual environment, or projected onto one or more of the walls of the virtual environment.

[0033] Embodiments may also include selecting an intervention based on one or more user profiles. The user profile may include one or more user settings that correlate the context scenario with preselected intervention criteria.

[0034] In a further embodiment, the user profile includes data related to the user's learning style, and the intervention includes tailoring the presentation of data to the user based on the user's learning style to facilitate learning. In yet a further embodiment, the platform is configured to read user feedback and tailor the presentation of data based on the user feedback. In other embodiments, the platform is configured to categorize users having similar learning styles and accordingly cluster data for presentation to users under the same categorization to facilitate group learning.

[0035] Further embodiments may include a plurality of client-side video conferencing applications and an interactive video conferencing platform configured to receive one or more of the video or audio streams from the plurality of client-side video conferencing applications. The interactive video conferencing platform may analyze the one or more video or audio streams and detect a context scenario based on the analysis. These embodiments may also include an intervention database including a plurality of interventions, and based on detecting the context scenario, the embodiments may retrieve and implement an intervention corresponding to the context scenario. These embodiments may also include an output signal including an output audio signal or an output video signal, and the interactive video conferencing platform is configured to modify the output signal based on the intervention corresponding to the context scenario. The intervention may include a change to one or more camera views and an environmental change.

[0036] The embodiments may also include various non-transitory computer-readable media including instructions executable on a processor. One instruction may be to connect a plurality of users to a virtual conference based on a plurality of requests to join the virtual conference, each session including one or more video or audio streams, which collectively form a plurality of video or audio streams. A further instruction may be to analyze the plurality of video or audio streams to detect a context scenario from a plurality of context scenarios stored in a scenario database. The instructions may also include instructions for automatically detecting a context scenario of a plurality of context scenarios stored in the scenario database. After detection, the media may select an intervention from an intervention database based on the context scenario. After selection, there may be instructions for retrieving an intervention from the intervention database based on the context scenario. Then, after retrieving the intervention, the media may include instructions for intervening in the virtual conference based on the intervention retrieved from the intervention database, the intervention including at least one change to the output audio signal or the output video signal.

[0037] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary aspects of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. [Brief description of the drawings]

[0039] [Figure 1] FIG. 1 illustrates an overview of an exemplary environment in which the systems, methods, and other aspects of the present disclosure may be implemented. [Diagram 2] FIG. 2 illustrates a general example process flow 200 of the present systems and methods. [Diagram 3]FIG. 1 illustrates an exemplary process for performing an intervention according to some embodiments. [Figure 4] 1 is an exemplary flowchart of an exemplary method of using artificial intelligence to perform an intervention in a computing environment, according to an aspect of the present disclosure. [Diagram 5] FIG. 1 illustrates an exemplary model for using artificial intelligence in a computing environment, according to aspects of the present disclosure. [Figure 6] FIG. 1 illustrates an exemplary virtual broadcast environment according to aspects of the present disclosure. [Figure 7A] FIG. 2 illustrates exemplary inputs and outputs according to an aspect of the present disclosure. [Figure 7B] FIG. 11 illustrates additional exemplary inputs and outputs according to aspects of the present disclosure. [Figure 8] FIG. 8 illustrates several potential AI camera scenarios 800 of certain embodiments of the present disclosure. [Figure 9] 1 is an exemplary flow chart according to certain embodiments of the present disclosure. [Figure 10] FIG. 1 illustrates an exemplary environmental intervention flow in accordance with certain embodiments of the present disclosure. [Figure 11] FIG. 1 illustrates an example system of components capable of executing instructions for the methods of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] In the following embodiments, systems and methods for creating more engaging virtual conference presentations are described. Current virtual conferences are relatively static and boring, either providing a single video feed of a speaker or providing multiple slots with several attendees all at once. The embodiments of the present disclosure may enable a variety of camera effects or other interventions to create more engaging presentations.

[0041] The subject matter of this description will now be described more fully hereinafter with reference to the accompanying drawings, which form a part of this description and which show, by way of illustration, certain exemplary embodiments. An embodiment or implementation described herein as "exemplary" should not be construed as preferred or advantageous over other embodiments or implementations, for example, but rather is intended to reflect or indicate that the embodiment(s) are "exemplary" embodiment(s). The subject matter can be embodied in a variety of different forms, and as such, the subject matter as is claimed or claimed is not intended to be construed as being limited to any exemplary embodiment described herein, which exemplary embodiments are provided for illustrative purposes only. Likewise, a reasonably broad scope of the subject matter as is claimed or claimed is intended. For example, the subject matter can be embodied as a method, device, component, or system, among other things. Thus, the embodiments can take the form of, for example, hardware, software, firmware, or any combination thereof (excluding software itself). As such, the following detailed description is not intended to be taken in a limiting sense.

[0042] Throughout this specification and the claims, terms may have a nuanced meaning beyond that explicitly stated, as suggested or implied by the context. Similarly, the phrase "in one embodiment" as used herein does not necessarily refer to the same embodiment, and the phrase "in another embodiment" as used herein does not necessarily refer to a different embodiment. For example, it is intended that the claimed subject matter includes, in whole or in part, a combination of the exemplary embodiments.

[0043] The terms used hereinafter may be interpreted in their broadest reasonable manner, even if the terms are used in conjunction with the detailed description of certain specific examples of the present disclosure. Indeed, certain terms may even be emphasized hereinafter, but any terms intended to be interpreted in any limited manner will be expressly and specifically defined as such in this detailed description section. Both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the features claimed.

[0044] In this disclosure, the term "based on" means "based at least in part on." The singular forms "a," "an," and "the" include plural referents unless the context indicates otherwise. The term "exemplary" is used in the sense of "example" and not "ideal." The term "or" is intended to be inclusive and means either, any, some, or all of the listed items. The terms "comprises," "comprising," "includes," "including," or other variations thereof are intended to cover a non-exclusive inclusion, whereby a process, method, or product comprising a list of elements does not necessarily include only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Relative terms such as "substantially" and "generally" are used to indicate possible variations of ±10% of the stated or understood value.

[0045] Referring now to the accompanying drawings, Figure 1 illustrates an overview of an exemplary environment (or system(s)) 100 in accordance with one or more embodiments of the present disclosure. Environment 100 may include, for example, first user device(s) 110 and second user device(s) 120, which are configured to communicate with network system(s) 130 and collaborative interactive video conferencing platform 140. Although two user device(s) 110 and 120 are shown in environment 100, additional user devices may be provided in environment 100 for communicating, synchronously or asynchronously, with network system(s) 130 and / or collaborative interactive video conferencing platform 140, and for participating in collaborative content creation of multiple users, in accordance with the present disclosure.

[0046] According to aspects of the present disclosure, the network system(s) 130 may define one or more networks, including wired or wireless networks. The network system(s) 130 may include, for example, the Internet and / or one or more cloud networks. Furthermore, the network system(s) 130 may include a public network, such as the Internet, a private network, such as an intranet, or a combination thereof, and may utilize various currently available or later developed networking protocols, including but not limited to TCP / IP-based networking protocols. The network system(s) 130 may be configured to communicatively couple the user device(s) 110 and 120 to the collaborative interactive video conferencing platform 140 to enable communication of data between the user device(s) 110 and 120 and the collaborative interactive video conferencing platform 140. The network system(s) 130 may generally be enabled to employ any form of computer-readable or machine-readable media for communicating information from one device to another. The network system(s) 130 may include communication methods by which information may travel between computing devices. The network system(s) 130 may be considered to be public or private network connections and may include, for example, a virtual private network, or encryption or other security mechanisms employed on the public Internet, etc.

[0047] In one embodiment, user device(s) 110 and 120 may communicate with collaborative interactive video conferencing platform 140 directly or indirectly through network system(s) 130 or other available communication channels. In cases where user device(s) 110 and 120 communicate directly with collaborative interactive video conferencing platform 140, collaborative interactive video conferencing platform 140 may be implemented and configured to facilitate communication via one or more communication methods described in connection with network system(s) 130 above, for example.

[0048] According to aspects of the present disclosure, the collaborative interactive video conferencing platform 140 may include server system(s) 142, video storage system(s) 144, neural network 146, virtual environment system(s) 148, and intervention database 150. In some embodiments, the interactive video conferencing platform 140 may be one or more servers that may be configured to perform some or all of the functions of the server system(s) 142, video storage system(s) 144, neural network 146, virtual environment system(s) 148, and intervention database 150 according to aspects of the present disclosure. The system(s) in this disclosure may include various implementations that may include various electronic and computer systems. One or more implementations described herein may implement functions using two or more specific interconnected hardware modules or devices with associated control and data signals that may be communicated between and through the modules, or as part of an application specific integrated circuit. Thus, the system(s) encompass software, firmware, and hardware implementations.

[0049] According to aspects of the present disclosure, the server system(s) 142 may include one or more data servers or databases that may be located on a cloud server / network, an edge server / network, within the network system(s) 130, and / or where the server system(s) 142 may be directly or indirectly integrated with the collaborative interactive virtual conference platform 140. The server system(s) 142 may store and process data received from the user device(s) 110 and 120 and / or the network system(s) 130 to generate engaging virtual conferences in accordance with embodiments of the present disclosure. Additionally, the server system(s) 142 may receive and transmit data or command signals to and from the video storage system 144, the neural network 146, the virtual environment system(s) 148, and the intervention database 150 to facilitate the execution of engaging virtual conferences.

[0050] According to aspects of the present disclosure, the video storage system(s) 144 may receive one or more inputs or commands from the first user 112 and the second user 122 through their corresponding user devices 110 / 120. As described above, the environment 100 may include more than two users or more than two user devices 110 and 120. Thus, the video storage system(s) 144 may receive inputs and commands from more than two users or more than two user devices. Alternatively, the video storage system(s) 144 may receive one or more inputs or commands only from the first user 112 or the second user 112 depending on the type of collaborative session. That is, the collaborative interactive video conferencing platform 140 may facilitate one or more virtual conferences.

[0051] In one embodiment, the first user 112 and / or the second user 122 may input commands synchronously or asynchronously to the first user device(s) 110 and the second user device(s) 120, respectively. The video storage system(s) 144 may receive the input commands from the user device(s) 110 and 120 directly or indirectly via the network system(s) 130 and / or the server system(s) 142. The video storage system(s) 144 may then transcribe or convert the input commands into text data. For input commands that may be received in the form of text data, conversion to text data by the video storage system(s) 144 may not be necessary. The video storage system(s) 144 may then analyze the converted or received text data depending on the type or form of the input command by capturing the semantics of the text data (e.g., understanding the context of the conversation), detecting duplications and / or redundancies (e.g., unnecessary information), labeling the text data, and / or detecting prompts. The video storage system(s) 144 may then clean or modify the analyzed text data, if necessary, for example, by removing detected duplications and / or redundancies. The video storage system(s) 144 may then proceed to extract prompts, if any, from the analyzed and cleaned text data. In some embodiments, the video storage system(s) 144 may utilize the trained machine learning model. The video storage system(s) 144 may then transmit the extracted clean prompts to the virtual environment system(s) 148. In some embodiments, the conversion or transcription of data may occur on the user device(s) 110, 120.Thus, the user device(s) 110, 120 may implement computer code that enables conversion of audio or other types of data into text and transmit the transcribed text to the collaborative interactive video conferencing platform 140. Alternatively, as described above, the transcription or conversion of data can be performed within the collaborative interactive video conferencing platform 140 by the server system(s) 142, such that the input device(s) 110, 120 may not need to perform any transcription or conversion of data. Alternatively, all of the functions of the server system(s) 142 may be performed by the neural network 146, including the transcription or conversion of text, parsing of input or command data, and / or cleaning of input or command data.

[0052] In one embodiment, the labeling process may be performed by the video storage system(s) 144 on any type of input or command data, such as image data, video data, and / or other types of audio data. For data that does not include text data, the labeling process may be considered a type of transcription of commands into text. For example, if the tone of a user 112, 122 is associated by the video storage system(s) 144 with a particular emotion signal, the video storage system(s) 144 may generate labels including, for example, angry, happy, sad, worried, enthusiastic, etc., which may be identified by one or more neural networks 146. Multiple participants may exhibit similar emotions, different emotions, or even conflicting emotions. The video storage system(s) may identify real-time emotions for each participant based on a set of training data.

[0053] According to aspects of the present disclosure, the virtual environment system(s) 148 can analyze certain factors, such as the tone of voice, words, and actions of participants, and can make certain interventions in the virtual meeting to keep the virtual meeting running smoothly or to achieve some other objective defined by one or more users.

[0054] According to aspects of the present disclosure, the virtual environment system(s) 148 may facilitate the generation of one or more computing environments that users 112, 122 in the environment 100 may utilize to collaborate to create content. The computing environments may include, but are not limited to, for example, augmented reality environments, virtual reality environments, and two-dimensional (2D) or three-dimensional (3D) simulated environments. Examples of computing environments and user collaboration in computing embodiments are also disclosed in co-pending U.S. patent application Ser. No. 17 / 006,327, which is incorporated herein by reference in its entirety. Additionally, any of the machine learning algorithms disclosed in that co-pending application may be incorporated for use individually or in combination with the machine learning models used by the video storage system(s) 144 and / or the virtual environment system(s) 148.

[0055] The system may allow a user to configure various possible outputs depending on the input received. The system may further use presets, or context scenarios, to allow programmatic selection of the output and / or to modify the output. Various context scenarios may include a business meeting in a small office or a large conference room, a birthday party at home, a doctor's office, or a nightclub.

[0056] Therefore, the system utilizes multiple data points available from many data sources, including user and non-user data sources. User data sources include data generated by user avatars, such as graphics data of their avatars, their avatars' positions and orientations, their avatars' voices, movements, activities performed by the user avatars, etc. Non-user data includes objects in the virtual environment, ambient noise, ambient music, lighting, brightness, colors, etc. The data can be processed to create camera outputs representing further stimuli for the user to generate emotions and actions. The processing of the data includes logical interpretation of the inputs by a neural network implemented by the system, which can be used to create corresponding camera outputs. The generation of further stimuli can help access a wider range of human emotions of users interacting in the virtual environment, which can enhance the user's motivation, empathy, learning, connection, and overall energy to create a more interesting and enjoyable virtual event, potentially increasing attention span, learning ability, and productivity. The systems and methods of the present disclosure can be used for any type of virtual event held in a virtual environment, such as virtual work meetings, video games, sales meetings, virtual conferences, seminars, etc.

[0057] The system may implement machine-learned empathy that enables a smart, empathetic virtual environment to generate interventions such as changes to one or more camera views or environmental changes. In this disclosure, the term empathy refers to the ability of a computer system implementing artificial intelligence algorithms to logically understand a user's mental perspective, emotions, and behaviors and use that logical understanding to create outputs that affect the user.

[0058] "Empathy" as employed in this disclosure can be divided into three categories. · Cognitive empathy: The ability to understand a user's thoughts, intentions, and motivations, and to influence their behavior through one or more interventions. Emotional empathy: The ability to understand emotions and moods. This can be based on subtle, non-verbal messages such as hand gestures, tone of voice, speaking or typing speed, pauses between words and / or sentences, facial expressions, hand gestures, etc. Social empathy: The ability to cognitively and / or emotionally understand a group and influence user behavior through one or more interventions. The system can read various cognitive and emotional signals from several types of inputs, identify the group's dominant perspectives, emotions, and behaviors, and define an intervention that can create a desired output. The system can further categorize and differentiate between interventions based on their socially specific cognitive and emotional dominance, for example, by providing a certain type of camera output for one group where one feeling is dominant, and another camera output for another group.

[0059] Embodiments may use empathy detection to generate interventions to create a positive virtual meeting experience.

[0060] Exemplary interventions may be, for example, turning up the volume of one user and turning down the volume of other users, putting a spotlight on one user, modifying the weather in the virtual environment, changing the lighting and / or color of the virtual environment, generating a popup with text prompting the user to take an action, generating audio or haptic data prompting the user to take an action, modifying the viewpoint of one or more user avatars in the virtual camera, etc. The resulting user actions can be further captured by the system and used as new inputs to generate corresponding new outputs.

[0061] The camera output may take one or more of the following forms: Indirect interventions: These are changes in color, sound, atmosphere, etc. that do not directly tell the user what type of behavioral change is expected from them. For example, changing the background color, shining a spotlight on someone, changing the music or weather are all indirect interventions.

[0062] Direct intervention: a text, visual, or audio output directing the user to do something, e.g., ask this question, provide them with additional data, provide a summary of the information, analyze the information, etc. This is related to explicit feedback, not something that the user would have to interpret, but linked to an explicit rule, e.g., if there is a user who has not spoken for a while, there could be a text prompt prompting the user to ask a question. In another example, a salesman is making a sale to a client in a sales meeting, and the client has not spoken for a long time. A time rule can be configured in the system, whereby if the system measures a long silence of the customer, the system can prompt the salesman for a direct intervention to prompt the salesman to ask a question. Such a time rule can be configured such that the system sends the prompt after a predefined number of seconds or minutes, and the time rule can be further influenced by the semantic context of the conversation and / or other factors. For example, if the previous conversation included an instruction from one user to another to give them five minutes to review a document, the time rule would take that time into account and not cause any interruption in between. The prompt can be in the form of a text, tactile, or audio prompt. Text is the default, but other means are possible as well.

[0063] Large-scale behavioral interventions: These interventions can be applied for video games or other applications that require group interaction. For the case of video games, large-scale behavioral interventions can prompt players to change the scenario, or make the world attack with meteors so that players go somewhere else, or make monsters appear so that players move elsewhere. These interventions can be applied for example to meetings, where if things are getting heated during a meeting, a large-scale intervention can prompt everyone to take a break. Such interventions offer no other option than to perform the prompted action.

[0064] 2 shows a general example 200 of a process flow for the present system and method. The system may have multiple inputs 205 corresponding to system inputs 290-299. System inputs 290-299 may be, for example, video or audio streams, typing speed, typing volume (cancellation), hand gestures, amount of speech time, facial (micro) expressions, mouse / swipe speed, geographic location, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, speech toxicity, device or speech cadence, bandwidth, throughput, and incoming signal strength.

[0065] The system may use multiple neural network cadences 210 (e.g., facial recognition, sound analysis, etc.) corresponding to neural networks 270-285 that process one or more input categories (e.g., sound 295, video 299, system input 290, etc.) depending on the system's configuration and the system inputs 290-299 received, and generate various forms of network data 215 corresponding to network outputs 265 and 260. The neural networks 270-285 may operate at different neural network cadences 210. For example, parallel input streams may be processed at different intervals or tick rates depending on what those streams are capturing. Emotions may be every 5-10 seconds. Movements may be processed several times per second. This may also vary depending on who is the center of the conversation. For example, the speaker may be processed more frequently than someone who is not an active participant. Each parallel input stream may be processed differently depending on the people generating the input(s), what they are doing, and other factors. These network outputs 260 and 265 are further processed by executing engine code 220 in the form of stored rule-based operations or logic tests 255 to generate one or more corresponding camera outputs. The camera outputs may be presented in the virtual environment on one or more surfaces thereof, for example on a virtual screen of the virtual environment on which the event is being broadcast. The camera outputs may take the form of one or more of the environmental changes 225, which correspond to the environmental changes 235 and 240. Live, uncensored camera feeds, - Near real-time camera view adjustments, or Replaying scenes, for example to produce a "best of" that contains the most relevant scenes of an event.

[0066] The system may then implement environmental changes 235 and 240 by making state changes 230 corresponding to elements 250 and 245, respectively.

[0067] Embodiments may include a processor configured to receive data from multiple sources, such as cameras, microphones, sensors, and other suitable input devices. The processor may then pre-process the data by applying filters to transform the data into an appropriate format for analysis. In some embodiments, the data may be segmented into smaller portions or frames to improve the accuracy of the analysis.

[0068] The preprocessed data can then be passed through a Convolutional Neural Network (CNN), a type of deep learning algorithm that is particularly suited to image analysis. The CNN can be trained using a large dataset of labeled images corresponding to various emotional states. During the training process, the CNN learns to recognize patterns in the data that are associated with specific emotions. The trained CNN can then be used to classify new data based on the patterns that it has learned.

[0069] In some embodiments, the system can use multiple CNNs, each trained on a particular type of input. For example, one CNN can be trained on facial expressions, while another can be trained on vocal intonation. The outputs of these CNNs can then be combined using a fusion algorithm to produce a more accurate prediction of the emotional state.

[0070] In addition to CNNs, other AI algorithms such as recurrent neural networks (RNNs) or support vector machines (SVMs) can be used depending on the type of input and the desired output. For example, RNNs can be used for time series data such as voice intonation, while SVMs can be used for hand gesture recognition.

[0071] Overall, the present disclosure provides methods and systems for using machine learning to detect emotions using a variety of inputs. The use of CNNs and other suitable AI algorithms allows for accurate and robust detection of emotional states, which may be useful in a variety of applications such as market research, mental health diagnosis, and human-computer interaction.

[0072] The camera output shown in the camera replay can be configured to prompt one or more behavioral modifications of one or more users in the virtual environment to affect the mood and dynamics of the event, which can create a more active, spontaneous, and realistic user experience. The output of the camera adjustments can include, for example, zooming in or out on the user, panning the camera, changing the camera angle, adding camera effects, modifying the camera field of view, etc. The resulting user actions can be further captured by the virtual camera and used as new inputs to generate corresponding new outputs.

[0073] Audio input may be captured by the user's microphone and may include the user's voice data, the music of the virtual environment, typing sounds, tone, and voice speed, speaking rhythm, prosody, etc. Video input may be captured by real and virtual cameras and may include virtual cutouts of various users and elements in the virtual environment along with their positions and orientations, the current viewpoint of each user's virtual camera, the users' micro facial expressions, hand gestures, head movements, and system input may be captured by the processing system of the virtual environment and may include typing speed, loading speed, frames per second (FPS), mouse / swipe speed, current system load, system capacity including available bandwidth, available computing power, available memory, etc.

[0074] Neural networks can use appropriate AI algorithms that take in data, such as visual data, and run it through a logical graph where weights are applied and a numerical response is returned for some attribute. There are a few permutations between AI algorithms that can be used depending on the type of input and the desired output. An example of one such AI algorithm is a convolutional neural network.

[0075] The network outputs may be one or more categories of processed inputs that are further processed by executing rule-based operations stored in the form of logical tests to generate one or more corresponding camera outputs.

[0076] The camera output, also referred to herein as an intervention, can be configured to prompt one or more behavioral modifications of one or more users within the virtual environment to affect the mood and dynamics of the event, which may create a more active, spontaneous, and realistic user experience.

[0077] The camera output may be personalized differently for each user based on the user profile. The user profile may include data including the user's background, preferences (e.g., topics of interest, music, food, etc.), culture, career, age group, social class, personality, learning type, etc. Preferences may also include preselected intervention criteria, such as whether the user prefers heavy or light interventions, choosing between specific interventions based on different context scenarios or emotional signals. The system may generate further user profile data that is used to continuously update the user profile based on the interactions the user has in the virtual environment. The system is therefore continuously learning from each user and uses the data to further improve itself, to better understand each user, and to increase its own empathy levels. Hence, multiple "best of" replays can be generated based on the user profile, with at least one possible replay being provided to each user.

[0078] In addition to replays, embodiments may generate an automated summary of the virtual meeting based on inputs such as detected emotion signals, body language, etc. The summary may be smart analytics indicating one or more meeting outcomes, such as whether the meeting was happy, sad, enthusiastic, bad, or resulted in a sale. The automated summary may also include a log of whether there were many action points, or statistics such as the amount of laughter, the number of questions asked, or the participation of various people in the virtual meeting. Embodiments may be directly connected with social media to automatically report the automated summary to social media platforms.

[0079] Indirect environmental interventions can be presented to users at an individual or group level, and therefore, depending on the desired outcome, the system can decide that certain interventions should only be shown to certain users and not others.

[0080] In an example of indirect intervention, if a user has not spoken in a meeting for a while, the intervention could be to play a personalized song that is known to be the user's favorite song, and that song could be played only to that user to encourage the user to be energized and re-engage in the conversation. In another example, if the system identifies that the energy of the virtual meeting is irritable, for example due to a majority of users showing signs of irritation (e.g., facial expressions, tone of voice, hand gestures, words used, etc.), the system could decide to add more calming colors, e.g., blue, green, or purple, into the virtual environment and / or play soothing background music in the virtual environment that diverts the energy of the virtual session. The songs and / or colors played could be individually selected based on the user profile, so each user could end up seeing and hearing something different, even if similar interventions (e.g., modifying colors and music) are selected. The system could further decide to modify the user's current virtual camera POV and / or display something in front of the virtual camera that could affect the user's emotions.

[0081] Similarly, direct interventions can be sent to multiple users simultaneously, with each intervention including differences based on the user profile.

[0082] The same logic applies to large-scale behavioral interventions. For example, if a game level change is to be made, the system can display a message to all users to proceed to move to a particular area, where the text of the message has a different color for each user based on the user profile. Additionally, the music played can be different for each user based on the profile.

[0083] In one embodiment, the virtual environment can be a 3D board game. The interventions can be large-scale, such as the board exploding. Based on the detected inputs, such as behavior, actions, emotions, etc., several different interventions can be made to the game. This can have a huge impact on the interactivity, retention, and churn of the product.

[0084] In other embodiments, such as training, sales, or educational environments, combining the power of a 3D environment with video conferencing services can be very useful. Often exciting graphics and light shows are very expensive to produce. They can be significantly cheaper in a virtual environment and achieve the same effect.

[0085] 3, by taking in some type of input 300 to identify existing behaviors 305, processing that data (through a neural network), and analyzing and evaluating the network output through logical tests 310, the system can design interventions 315 in the form of camera output that are intended to prompt different behaviors 320 in the virtual environment. Those interventions can be determined based on learned empathy, as described throughout this specification.

[0086] 4 illustrates an example flow chart 400 in which system input 405 captured by microphone 475 and webcam 480 is sent for processing by four neural networks 410, each configured to process a different element, such as head movement 455, hand movement 460, emotion 465, and gesture 470 (including their position and orientation in the virtual environment). Neural networks 455, 460, 465, and 470 may periodically (e.g., every second) poll the input data and generate network data 415, such as head position 445 and tone of voice 450. Polling may occur at different neural network cadences 410 based on a number of factors, such as who is speaking, the need for frequent sampling, bandwidth, and other computational resources. Cadence changes may occur in real time. The selected network data 415 can be used in a logic test 440 to conclude an emotion such as anger, which may prompt a corresponding output 425 in the form of an increase in FOV 430, a change in camera angle, moving the camera further away from or closer to the user, changes in weather, sounds, colors, etc. For example, if the tone of voice 450 is elevated and the head position 445 (and possibly facial gestures) are characterized by the system as somehow "aggressive," the logic test 440 may conclude, based on the training of the neural network 410, that the user is exhibiting anger.

[0087] The output may be in the form of a change in the weather outside the virtual environment, for example, starting a thunderstorm, playing the sound of thunder, amplifying the user's voice, changing the background color (e.g., to red), or fading the color, such as from red to purple or blue. The environmental intervention may thus prompt behavior and / or emotion. The behavior is further sensed by the virtual environment to generate further output as necessary.

[0088] In some embodiments, the system allows a recording of an event in the virtual environment to be presented to the user if desired. In other embodiments, the system further allows applying AI algorithms that allow summarizing the event to select the most interesting scenes of the event and create a summary thereof. The recording may be done through one or more virtual broadcast cameras at the event capturing a wide range of data including user and non-user data. The event recording may be presented on one or more surfaces of the virtual environment or may be stored in memory for later transmission to a user device for playback by the user at a later time. The event recording may be further processed, if necessary, before being sent to the user. The event recording may be detected, for example, by identifying strong reactions from the participants. The reactions may be stored using several camera angles at a time and stored in memory for later playback, such as after the virtual meeting is over, at the end, or at any time the user requests playback. The playback may be on the entire user screen or on a surface of the virtual environment, such as a virtual whiteboard.

[0089] The most important parts for playback can also be based on context scenarios: for example, a business meeting may need to be a selection of key points, which can be detected via a combination of speech recognition and empathy trained neural networks, or a meeting on a recreational subject would have a different context, such as identifying the activities that generated the strongest reactions or laughs.

[0090] FIG. 5 shows an exemplary model 500 in which the system receives system inputs 505 such as sound inputs 590 including position inputs 592 of people making sounds in the virtual environment. The sounds and positions are processed at 510 and aggregated at 588 to extract various sound parameters 515 such as identifying who is speaking 582, what is their tone of voice 584, the length of their speech 586, and the recipient of their message 587. For each sound parameter 515, there can be a respective response 520, which in this example are "Alex is speaking" 565, "his tone is calm", "he has been speaking for 10 seconds", and "he is speaking to Bill". For such sound analysis, a spectrogram of the sound can be used to provide valuable data that can be processed to extract the required response. Spectrogram analysis focuses on the energy, amplitude, or frequency of someone's voice and then tries to analyze and estimate the magnitude of its curve or pace of camera changes 540 and 545 and predict the pattern of moving the camera accordingly. This data can be passed through engine code 525, such as rule-based logic tests 560, to determine one or more camera changes 530, such as moving the camera 550, or modifying the cutting speed and duration of a camera take 555. For example, a rule might be that after 10 seconds of the same person speaking, there should be a camera change 540, such as a camera cut, or another camera change 545, such as a slow pan to another speaker to prompt a change of speaker, for example, prompting Bill to interrupt Alex and ask a question.

[0091] The system may be trained to improve user experience based machine learning using its own feedback data or user preferences or input. For example, if after one or more rounds the system thinks that it is too early to perform a cut after 10 seconds of speech, the system may add more seconds to the next round. The system may further perform semantic analysis and processing to identify what is being said in the event and generate the camera output accordingly, for example, if a user says "I'm going to wrap up this meeting," the framing, lighting, cut rate, etc. of the camera output may be modified to create more dramatic effects or environmental changes, such as adding epic music and / or dimming the lights in the virtual environment. The system may further use direct user feedback to improve itself. There are various algorithms that may be trained to summarize data, such as those used by programs including Blinkist® and Newsleo®, which may be incorporated into the system.

[0092] The embodiment may solicit various types of feedback data before, during, and after the virtual meeting. This data may then be used to train a neural network to detect what people did or did not enjoy and modify their behavior based on that information. The feedback may be based, for example, on a slider included in the user interface (UI) that indicates that the user is enjoying the virtual meeting. The system may prompt the user to move the slider at various times, such as when an intervention occurs, or periodically, such as every 5 minutes, or a combination of the two. In this manner, the system may be able to identify the difference between successful and unsuccessful interventions based on user rankings that correspond to the position of the slider for each individual user. Alternatively, the user may be asked to provide a number from 1 to 5, or a value such as "good," "fair," or "bad." Moreover, the system may be able to identify various subtleties of intervention. Some cases may require a gentle intervention, such as an implied pause in conversation. Other cases may require a more significant intervention, such as an argument or fight and the meeting needs to calm down. The selection of which intervention to use is discussed further below.

[0093] The system may further apply further processing, such as pre-smoothing, to the spectrogram data to predict where someone, for example, will move their mouth next. In this context, for example, if someone is going to make a statement that will last, for example, five seconds, it can be set up on the camera and moved accordingly, or environmental changes can be made to the virtual environment. In this manner, the system can capture the emotional energy of the meeting or event and adjust the camera or modify the virtual environment accordingly based on this data.

[0094] Preemptive smoothing can be used for 2D motion information, such as moving puzzle pieces ahead of time, but there is more that can be done with it, for example, if someone is tracking their fingers, that is a lot of data, and therefore if it can be interpolated and smoothed, the FPS can be increased. The embodiments can increase the FPS by reducing the granularity of fine features such as fingers, allowing for higher throughput and less bandwidth consumption. This can be advantageous, especially at scale, because there may be hundreds or thousands of virtual meetings occurring in parallel. Saving a little bit of data for each of those ongoing virtual meetings can save a lot of aggregate computer and network resources.

[0095] FIG. 6 shows an overall system 600, where multiple virtual broadcast cameras 670 are placed in the virtual environment to capture different angles that can be used to capture more details from the scene, and multiple microphones capture sounds from within the scene, which can be inputs and outputs to drive corresponding processing 635. The various camera feeds can also be used to generate more possible data for processing 635 to generate more possible replays, define live camera adjustments, or make environmental changes to drive behavioral changes. The system further captures the spatial position or location 665, orientation, speech content 660, and state of each of the objects and avatars in the virtual environment, for example, of a virtual meeting, such as focusing on heads 640, hands 645, emotions 650, and gestures 655, all of which can be reconstructed 605 from any angle to generate corresponding replays. The spatial position, orientation, and state can be metadata of the input data and can therefore be used to further characterize the input data to derive different types of processing 635 and outputs. The resulting camera output may be, for example, various types of camera framing 620, camera motion 625, cutting 630, environmental changes 680, etc. The system further captures speech content that may be used for semantic analysis 615 that may provide further feedback to generate the resulting output.

[0096] The present disclosure provides systems and methods for capturing video using various camera angles to enhance the visual storytelling experience. In particular, the present disclosure describes various camera angles, including eye-level, Dutch, low-angle, high-angle, top-angle, and over-the-shoulder, that can be used to capture video in a unique and compelling way.

[0097] An eye-level camera angle refers to the placement of a camera at the same height as the subject's eyes. This angle is often used in interviews and dialogue scenes to create a sense of intimacy and connection between the subject and the viewer. By using an eye-level camera angle, the viewer feels as if they are in the same virtual environment as the subject and are having a conversation with them.

[0098] Dutch camera angles, also known as oblique angles, refer to tilting the camera to create a slanted or diagonal composition. This angle can be used to create a sense of tension, anxiety, or disorientation in the viewer. Dutch angles are often used in horror or thriller films to create a sense of anxiety or disorientation in the viewer.

[0099] A low-angle camera angle refers to a camera positioning below eye level and looking up at the subject. This angle may be used to create a sense of power, superiority, or bravery in the subject. Low-angle shots are often used in action movies to make the protagonist appear larger than life.

[0100] A high-angle camera angle refers to the placement of the camera looking down, above eye level of the subject. This angle can be used to create a sense of weakness or helplessness in the subject. High-angle shots are often used in horror films to create a sense of vulnerability in the victim.

[0101] A top-angle camera angle refers to the placement of a camera directly above and looking straight down on a subject. This angle can be used to provide a unique and interesting perspective on a subject. Top-angle shots are often used in documentaries or nature films to show a subject from a different perspective.

[0102] An over-the-shoulder camera angle refers to a camera positioning behind one character looking over the shoulder of another. This angle can be used to create a sense of intimacy or tension between the characters. Over-the-shoulder shots are often used in dialogue scenes to show the reactions of both characters.

[0103] Overall, this disclosure provides systems and methods for capturing video using various camera angles to enhance the visual storytelling experience. By using various camera angles, filmmakers can create unique and compelling footage that captures the viewer's attention and helps tell the story in a more engaging and effective way.

[0104] An upshot camera shot refers to the positioning of the camera below and looking up at the subject. This shot may be used to create a sense of power or dominance in the subject. Upshot shots are often used in action movies to make the protagonist appear larger than life.

[0105] A drop shot camera shot refers to the placement of the camera above the subject, looking down. This shot can be used to create a sense of weakness or helplessness in the subject. Drop shots are often used in horror films to create a sense of weakness in the victim.

[0106] An over-the-shoulder camera shot refers to a camera positioning behind one character looking over the shoulder of another. This shot can be used to create a sense of intimacy or tension between the characters. Over-the-shoulder shots are often used in dialogue scenes to show the reactions of both characters.

[0107] A close-up camera shot refers to a camera placement very close to a subject, capturing a detailed view of the subject's face or body. This shot may be used to create a sense of intimacy or to emphasize the subject's emotions.

[0108] An extreme close-up camera shot refers to the placement of the camera very close to the subject, capturing small details such as the eyes or lips. This shot may be used to emphasize a particular emotion or feature of the subject.

[0109] A medium shot camera shot refers to a camera positioning that is a moderate distance from the subject, capturing the upper body or waist up. This shot is often used to show the subject's body language or physical interaction.

[0110] A long shot camera shot refers to a camera placement at a great distance from the subject, capturing the whole body and the surrounding environment. This shot is often used to establish a location or setting.

[0111] A single-shot camera shot refers to a camera arrangement that captures only one subject.

[0112] A two-shot camera shot refers to a camera arrangement that captures two subjects in a single shot.

[0113] A three-shot camera shot refers to a camera arrangement that captures three subjects in a single shot.

[0114] A point-of-view (POV) camera shot refers to a camera placement that captures a scene from a character's point of view. This shot can be used to provide the viewer with a unique, immersive experience, as if they were experiencing the scene firsthand.

[0115] The present disclosure provides systems and methods for capturing video using a variety of camera shots to enhance the visual storytelling experience. In particular, the present disclosure describes a variety of camera shots, including up shot, down shot, over the shoulder, close up, extreme, medium shot, long shot, extreme, single shot, two shot, three shot, and point of view (POV), that can be used to capture video in a unique, compelling, engaging, and engaging way.

[0116] 7A and 7B show various separate parameters 700 including inputs 705, presets 710, and outputs 715, 720 that may be combined as appropriate. The presets 710 are context scenarios into which the inputs are taken, such as the type of meeting the virtual conference is intended for, and the outputs 715, 720 are generated and may affect the type of logical tests performed on the inputs. The inputs 705 may include typing speed, typing volume (cancellation), hand gestures, amount of speech time, facial (micro) expressions, mouse / swipe speed, geographic location, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, language toxicity, device, or speech rhythm, etc. The presets 710 may be pre-selected by a meeting administrator and may further include the event category, for example, whether the event is a business meeting, a sales meeting, a classroom, a chat show, an Alfred Hitchcock, a talk show, a YouTube®, a slow pan, a jump cut, a Snowy cam, a predictive view, a Stanley Kubrick, a freehand, a TikTok®, a fast cut rate, a fade to black, a J / .L cut, a selfie cam, a workshop, or a stand-up, etc. For example, an input identified as an expression of joy may result in a different output 715, 720 in a workshop compared to a conversation between friends, in which case the replay, music, or environment may be set to be more formal in the workshop than in a friendly conversation. The outputs 715, 720 may include brightening / brightening, depth of field (DOF), bloom (shader effect), cutting style, different color settings, camera angle, particles, shading, cutting speed, 3D lookup table, camera movement, and camera shot size.

[0117] A boardroom is different from a showroom, for example, which is different from a retail store. Every virtual environment is different, and inputs may mean different things in each context and create different outputs 715, 720, so it's like a framework and affects both inputs and outputs. Presets can be chosen before the meeting and can be modified by the host. For a certain time frame, the presets are predefined settings, and then the dynamics of the event may change, or the host may step in and change the settings, or something may happen that changes the settings so much that it becomes a new context (like an output, etc.).

[0118] FIG. 8 illustrates several potential AI camera scenarios 800 for a particular embodiment. These may include situations 805 such as someone is speaking, a conversation between people, someone has not spoken for a long time, someone is speaking for too long, the meeting is ending, and someone is showing anger, etc. Data inputs 810 may include 1) speech, rhythm, position, 2) speech, rhythm, position of multiple people, 3) length of time between the last speech and now, 4) duration of speech, 5) duration of meeting, no more speakers left, 6) speech, rhythm, gestures, and tone, etc. Outputs 815 may include panning or zooming a camera to a speaker, multiple cameras switching between them and zooming in and out, focusing and zooming a broadcast camera on a participant, moving the camera away from a speaker to another speaker, zooming in to create a dramatic camera effect. Actions may include focusing on the person speaking, keeping attention on the people speaking, the user starting to speak, the user finishing speaking, or the user noticing a reaction and adjusting their actions, etc.

[0119] FIG. 9 shows a flow chart 900 according to one embodiment. This embodiment can be executed by an interactive virtual conference platform in one or more virtual environment systems and can include multiple virtual environments stored in memory. The platform can also store multiple context scenarios and emotional signals, as shown in step 902. In this embodiment, a virtual conference begins at step 904. This can include receiving one or more requests from one or more user devices to join the virtual conference. The system can then associate the one or more requests together based on a unique identifier or a set of pre-programmed values. The requests can be connected to form a virtual conference. At step 906, the system can analyze the received input data, including video and audio feeds from one or more user devices, looking for pre-programmed context scenarios or emotional signals, as further discussed above. At step 908, if the system detects such context scenarios or emotional signals by analyzing the input data, the system can select one or more interventions from an intervention database that are correlated to the context scenario and the input data. Such a selection may be influenced by the empathy level that the system has learned based on training data and user feedback. As discussed above, several inputs may be used to select an intervention, such as tone of voice, user preferences, and voice sound level. All this information may be used to identify, for example, that a conversation is angry and an intervention is needed to soften the conversation. The system may select an intervention from an intervention database or memory, which may be retrieved and applied in step 910. For example, a clown may walk across the screen or soothing music may begin to play. In step 912, the system may collect feedback to determine the success of the intervention. Success may be determined in a variety of ways.For example, if the goal is to de-escalate the conversation, the system may monitor the tone or words used to determine if the conversation has de-escalated. Or, for example, if the system is set to encourage arguments for entertainment purposes, success may be determined based on whether there is more anger in the conversation. Further feedback from the participants may be solicited via one or more user inputs. The user input may be on a scale such as 1-10 and may include items such as whether the user enjoyed the intervention, how effective the intervention was with respect to a given goal, and whether a different intervention would have been preferred. The system may then collect this feedback and incorporate this feedback into future virtual meetings to make future virtual meetings more successful. Finally, at step 914, the system may end the virtual meeting or repeat the process beginning with step 906 until the virtual meeting is over.

[0120] FIG. 10 shows an example environment intervention flow 1000. For example, if people are stuck in a virtual environment at 1005, the virtual environment may assess that they are stuck based on lack of verbal input or lackluster body language at 1010. The virtual environment generates an intervention such as a noise / object / character that brings about a shift in the group behavior at 1015. Behavior may be further measured and used as input at 1020 to determine if the intervention was successful or unsuccessful with a view to improving future interventions or modifying the intervention. For example, volume, color, tone, shape, etc. may all lead to various behavioral systems that the neural network may assess based on physical and verbal responses to the intervention(s).

[0121] 11 illustrates an example of a computing device 1100 of a computer system. The computing device 1100 may include a processor(s) 1110 (e.g., a CPU, GPU, or other processing unit), a memory 1120, and a communication interface(s) 1140 (e.g., a network interface) for communicating with other devices and receiving inputs from those devices, such as typing speed, typing volume (cancellation), hand gestures, amount of speech time, facial (micro)expressions, mouse / swipe speed, geographic location, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, language toxicity, device, or rhythm of speech, and the interactive video conferencing platform is configured to select an intervention based on one or more of those inputs. The memory 1120 may include volatile memory, such as RAM, and / or non-volatile memory, such as ROM and storage media. Examples of storage media include solid-state storage media (e.g., solid-state drives and / or removable flash memory), optical storage media (e.g., optical disks), and / or magnetic storage media (e.g., hard disk drives). The instructions (e.g., software or computer readable code) may be stored in any volatile and / or non-volatile memory components of the memory 1120. The computing device 1100 may further include input device(s) 1150 (e.g., keyboard, mouse, joystick, controller, or touch screen) and output device(s) 1160 (e.g., display, heads-up display, AR display, VR display, printer) in some embodiments. For example, if the user device(s) 110, 120 may be embodied as a tablet computer, the user device(s) 110, 120 may have a touch screen and a display.The aforementioned elements of computing device 1100 may be coupled together through bus 1130, which may represent one or more buses. In some embodiments, processor(s) 1110 of computing device 1100 include both a CPU and a GPU.

[0122] As an example, the system may detect that too many people are talking while some are quiet. The system may take in noise from multiple inputs, conclude that many people are active and one is passive (or silent), and prompt a decrease in volume for all active users and a spotlight on the silent user. As a result of this behavioral intervention, other users may speak less and the silent person may speak more, restoring balance to the meeting. Users may be able to set in their profile how aggressively they want the system to use this intervention for themselves or other users.

[0123] A face detection AI algorithm may be used to generate face geometry coordinates, which may create a steering vector for the user's head orientation. Two views are possible as the user turns their head.

[0124] First, a position vector can be detected by a virtual camera facing the user in 3D space, which can then be moved in a converse motion to preserve the illusion that the 2D camera feed is pointed correctly within the 3D geometry of a virtual meeting that may be in the virtual environment. This technique is sometimes called match moving. The user's head as a controller can also be applied to entities in the virtual environment, such as lights, to cast dynamic shadows as the user moves around.

[0125] Second, the position vector can be detected by a camera that represents either the user's first or second person perspective in 3D space. There can be scaling / limitation and minimum fence logic as the user moves their head left / right, up / down, or forward / backward. This can allow the 3D engine to draw geometry that is not literal. In the 3D space of virtual reality, the user's movements can be captured 1:1, while on a 2D screen the user's movements are scaled accordingly.

[0126] The hand pose may be used to map the coordinates of the user's fingers in 3D space. The 3D coordinates of each knuckle / finger may be captured essentially in real time. This information may then be passed to a rigged 3D model of the hand. The user's skin color may additionally be captured from a representative patch on the forehead identified by the neural pace detection network. This color / texture may then be mapped to the luminous properties of the texture of the rigged 3D model of the user's hand. This may then enable the virtual environment to display a personalized, realistic 3D hand. The hand may also be rigged for gestures / grabbing / pinching or other movements. The more gross movements of the hand may be mapped to any visual properties of the virtual environment, such as weather, lighting, wind outside, etc.

[0127] Additional problems that certain embodiments may address include a standardized educational environment that may slow down the learning of many students due to most people having different learning styles, personalities, abilities, etc. If the user is not emotionally connected to the content at hand, the user will become bored, which increases resistance to learning and stress among the students. Each user may have a predefined profile that may be based on preferences entered by the user or past experiences with the user's response(s) to various interventions. Thus, each presentation may be customized for each person. Some people may learn better in a more formal school-like virtual environment, while another person may learn better in an environment like a virtual park.

[0128] An embodiment may solve this problem by creating a smart virtual environment. In this embodiment, the system may be used to enhance learning in a classroom or seminar context. The user profile may include preferences, such as topics of interest, music, food, background images, culture, background, age group, social class, personality, learning type, etc. The system may generate further user profile data that may be used to continuously update the user profile based on the interactions the user has in the virtual environment and reactions to interventions. The system is therefore continuously learning from each user and using the data to further improve itself and understand each user better to increase its own empathy levels.

[0129] A user's learning style is also useful because it identifies how each user prefers to learn in order to increase the user's learning efficiency.

[0130] An administrator (e.g., a teacher) can define the content that will be delivered to a class. The system can analyze the content and each student's user profile and output multiple presentation media that best match each student's user profile. Thus, each student may receive separate presentations of similar information. By finding the presentation media that best matches each student, students can connect with their particular learning style and increase their emotional connection to the content, facilitating and enhancing their learning.

[0131] Processing the content taking into account the user profile can include summarizing the content in a relatively different perspective for each student, where for a particular student the system can use more emotive language or language that may resonate more with the student's particular profile, while for other students it can use language that is more technical and with a higher level of detail. The system can also generate or select examples that can illustrate a particular situation in a way that better resonates with the student. Furthermore, the type of visual media to be selected can vary from user to user. For example, if a student is defined as a visual learner, the system can create short movies and / or select or mix movies from various sources that can constitute a more efficient learning presentation for the student. For another user who is more analytical and prefers reading, texts or video clips with more analytical examples can be provided to that user. Modifications in the virtual environment can also be made individually, including changes in color, background, sounds, atmosphere, etc., that can be used to enhance each student's learning.

[0132] In some embodiments, the system may modify certain aspects of the content to be delivered based on reading the user input in substantially real time. For example, if the user appears confused or expresses doubts, the system may read this and ask the user what, if anything, can be done to improve the user's experience, providing options or leaving space for the user to describe their situation. When the system detects a change in the user's behavior, the system may further adapt certain parts of the content, visuals, audio, etc. in real time. The presentation may be in the form of video, audio, haptic experiences, a complete modification of a virtual environment to represent the lesson being delivered, which may take place in AR and / or VR, etc. The system may also be a program that provides these various types of content presentations, not necessarily virtual environments, from which one of them is for, but is not limited to, a presentation in a 3D virtual environment. The system may also provide this information to a teacher or parent to make additional interventions for the child to ensure that the child understands the material.

[0133] The system can further categorize students with similar learning styles and provide lessons in similar or the same media. This may also help to promote group learning and interaction, since users with the same learning style can benefit from each other's interpretations and processing of the lesson. Users can be categorized into one or more learning styles (e.g., visual or auditory) and receive similar virtual environments and interventions as users under the same categorization.

[0134] Instructions executable by one or more processors may be stored on a non-transitory computer-readable medium (computer-readable storage medium). Thus, whenever a computer-implemented method is described in this disclosure, the disclosure shall also be understood to describe a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, configure and / or cause the one or more processors to perform the computer-implemented method. Examples of non-transitory computer-readable media include RAM, ROM, solid-state storage media (e.g., solid-state drives), optical storage media (e.g., optical disks), and magnetic storage media (e.g., hard disk drives). A non-transitory computer-readable medium can be part of the memory of a computer system or can be separate from any computer system.

[0135] In the above description of exemplary embodiments, it should be understood that various features may be grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects reside in less than all features of a single foregoing disclosed embodiment. The use of the word "or" herein is intended to be non-exclusive and should be interpreted to include the individual elements alone or in any combination thereof. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present disclosure.

[0136] Moreover, although some embodiments described herein include some features included in other embodiments and not others, as would be understood by one of ordinary skill in the art, combinations of features of the various embodiments are intended to be within the scope of the disclosure and to form various embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0137] Thus, while particular embodiments have been described, those skilled in the art will recognize that other and further modifications may be made to those embodiments without departing from the spirit of the disclosure, and that all such changes and modifications are intended to be claimed as falling within the scope of the disclosure. For example, functionality may be added or deleted from the block diagrams, operations may be swapped between functional blocks, and steps may be added or deleted to methods described within the scope of the disclosure.

[0138] The subject matter disclosed above should be considered as illustrative and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present disclosure. The use of the term "or" herein means one or more of any of the available options, and does not necessarily mean a mutually exclusive list. Therefore, to the maximum extent permitted by law, the scope of the present disclosure should be determined by the broadest permissible interpretation of the following claims and their equivalents, and is not intended to be limited or restricted by the foregoing detailed description. Although various embodiments of the present disclosure have been described, it will be apparent to one of ordinary skill in the art that many more embodiments are possible within the scope of the present disclosure. Thus, the present disclosure is not intended to be limited.

Claims

1. storing, by the interactive virtual conferencing platform in one or more virtual environment systems, a plurality of virtual environments; storing, by the interactive virtual conference platform within the one or more virtual environment systems, a plurality of context scenarios; storing, by the interactive virtual meeting platform within the one or more virtual environment systems, a plurality of emotion signals; receiving, at the interactive virtual conference platform, a plurality of requests to join a virtual conference; connecting, by the interactive virtual conference platform, a plurality of sessions corresponding to the plurality of requests, each session including input data including at least one or more video or audio streams, the video or audio streams collectively forming a plurality of video or audio streams; analyzing, by a server of the interactive virtual conference platform, the plurality of video or audio streams; automatically detecting, by the interactive virtual meeting platform, a context scenario of the plurality of context scenarios or one or more emotion signals from the input data; selecting, by the interactive virtual meeting platform in response to detecting the contextual scenario or the emotional signal, an intervention from an intervention database based on the analyzed input data, the detected contextual scenario, or the detected one or more emotional signals; retrieving, by the interactive virtual conference platform, the intervention from the intervention database in response to detecting the context scenario; intervening, by the interactive virtual conference platform, in the virtual conference based on the intervention retrieved from the intervention database, the intervention comprising at least one change to an output audio signal or an output video signal; The method includes:

2. inputting, by the interactive virtual meeting platform, one or more data sets corresponding to the plurality of context scenarios into one or more neural networks; The method of claim 1 further comprising:

3. 3. The method of claim 2, wherein the one or more neural networks include one or more of a convolutional neural network (CNN) and a recurrent neural network (RNN).

4. receiving feedback about the intervention and applying the feedback to the one or more neural networks to train the one or more neural networks for application to future interventions; The method of claim 2 , further comprising:

5. The method of claim 4 , wherein the feedback comprises a ranking from at least one user.

6. The method of claim 5 , wherein the feedback comprises a physical reaction from one or more users detected via the video or audio stream.

7. The intervention, The method of claim 1 , including at least one of a change to a virtual camera angle, a change to a shot size, and a change to a camera motion.

8. The intervention, 10. The method of claim 1, comprising one or more of: modifying a brightness of the output video signal; modifying a tone of the output audio signal; and modifying a hue of the output video signal.

9. The method of claim 1 , wherein selecting the intervention further comprises selecting the intervention based on one or more user profiles.

10. The method of claim 9 , wherein the one or more user profiles include one or more user settings that correlate context scenarios with preselected intervention criteria.

11. A plurality of client-side video conferencing applications; an interactive video conferencing platform configured to receive one or more of a video stream or an audio stream from the plurality of client-side video conferencing applications, the interactive video conferencing platform configured to analyze the one or more video streams or audio streams and to detect a context scenario based on the analysis; an intervention database of the interactive video conferencing platform including a plurality of interventions, wherein the interactive video conferencing platform is further configured to retrieve and implement an intervention corresponding to the context scenario based on detecting the context scenario; and Equipped with the interactive video conferencing platform is configured to modify an output signal of the interactive video conferencing platform, including an output audio signal or an output video signal, based on the intervention corresponding to the context scenario. Computer system.

12. The computer system of claim 11 , wherein the interventions include a change to one or more camera views and an environmental change.

13. 13. The computer system of claim 12, wherein the interactive video conferencing platform is further configured to receive feedback about the interventions and to apply the feedback to one or more neural networks to train the one or more neural networks for application to future interventions.

14. a network interface configured to receive inputs including one or more of typing speed, typing volume (cancellation), hand gestures, amount of speech time, facial (micro) expressions, mouse / swipe speed, geographic location, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, language toxicity, device, or speech rhythm, and wherein the interactive video conferencing platform is configured to select the intervention based on one or more of the inputs.

12. The computer system of claim 11.

15. The intervention, modifying the brightness of the output video signal; modifying the tone of the output audio signal; and modifying the hue of the output video signal.

12. The computer system of claim 11.

16. the intervention corresponding to the context scenario further corresponds to one or more user profiles; 12. The computer system of claim 11.

17. The computer system of claim 16 , wherein the one or more user profiles include one or more user settings that correlate context scenarios with preselected intervention criteria.

18. A computer-readable storage medium having instructions stored thereon, the instructions are executable by a processor; The instruction: connecting a plurality of users to a virtual conference based on a plurality of requests to join the virtual conference, each session including one or more video or audio streams, the video or audio streams collectively forming a plurality of video or audio streams; analyzing the plurality of video or audio streams to detect a context scenario from a plurality of context scenarios stored in a scenario database; automatically detecting the context scenario from among the plurality of context scenarios stored in the scenario database; selecting an intervention from an intervention database based on the context scenario; retrieving the intervention based on the context scenario from the intervention database; intervening in the virtual conference based on the intervention retrieved from the intervention database, the intervention comprising at least one change to an output audio signal or an output video signal; A computer-readable storage medium comprising:

19. The instruction: analyzing one or more neural networks, including one or more of a convolutional neural network (CNN) and a recurrent neural network (RNN), to detect the context scenario; 20. The computer-readable storage medium of claim 18, further comprising:

20. The computer-readable storage medium of claim 18 , wherein the intervention corresponding to the context scenario further corresponds to one or more user profiles.

Citation Information

Cited By

  • system

    JP2026155728A

  • Webinar video generation system, webinar video generation method, and webinar video generation program

    JP7874369B1

  • Webinar video generation system, webinar video generation method, and webinar video generation program

    JP7897669B1