Intervention system and method in multi-modal data flow artificial intelligence virtual environment

By analyzing user emotional cues through an AI-powered camera system and adjusting the lighting and sound effects of the virtual environment, the problem of insufficient participant interest and interaction in traditional video conferencing is solved, resulting in a more engaging and interactive virtual meeting experience that improves collaboration and productivity.

CN118897637APending Publication Date: 2024-11-05TMRW FOUND IP & HLDG S A R L
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410601250.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-02
Filing Date
2024-05-15
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Traditional video conferencing results in a lack of interest and attention among participants, a lack of face-to-face interaction and non-verbal emotional cues, leading to difficulties in team collaboration and trust building. Existing technologies have failed to provide solutions for generating engaging virtual meetings.

Method used

By employing an AI-powered camera system based on multimodal data streams, the system analyzes users' emotional cues such as body language, facial expressions, and tone of voice to adjust the lighting, sound effects, and conversation rhythm of the virtual environment, thereby creating a more interactive and engaging virtual meeting experience.

Benefits of technology

It enhances the naturalness and spontaneity of virtual meetings, improves participants' attention span and motivation, promotes better communication and collaboration, and increases meeting productivity and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure HDA0004840642340000011
    Figure HDA0004840642340000011
  • Figure HDA0004840642340000021
    Figure HDA0004840642340000021
  • Figure HDA0004840642340000022
    Figure HDA0004840642340000022
Patent Text Reader

Abstract

Embodiments of the present disclosure include systems and methods for an improved video conferencing system in a virtual environment. Embodiments may use artificial intelligence, large language models, and machine learning to detect the mind and mood within a group of participants in a virtual conference. Various inputs, such as audio / visual streams, other sensor data, and training sets, may be used to detect various emotional scenes, such as boring, happiness, and sadness. Embodiments may detect such scenarios and take intervention based on desired behavior changes in participants, such as encouraging more participants or waking up people. The intervention may be a change in camera viewing angle or an environmental change in the virtual environment. The intervention may also depend on contextual scenarios, such as a severe business conference and a friend party.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to creating content in a computing environment, and more particularly, to systems and methods for enhancing video conferencing technology using artificial intelligence (AI). Artificial intelligence can develop a more engaging experience by generating a more interactive viewing experience that goes beyond a typical speaker grid or focuses on a speaker. Background Art

[0002] Video conferencing has become an essential tool for communication and collaboration in today's world. However, traditional video conferencing formats often result in unengaging and unproductive meetings, resulting in a lack of interest and focus among participants. Boring video conferences can lead to reduced collaboration and communication among team members, lowering productivity and efficiency in achieving meeting goals. Therefore, there is a need to improve video conferencing to make it more interesting and engaging for participants. This can be achieved by developing innovative technologies and features that enhance the user experience and encourage active participation, making meetings more productive and successful.

[0003] As remote work and virtual collaboration become more common, the limitations of traditional video conferencing formats have become increasingly apparent. During lengthy or unstructured meetings, participants often struggle to stay focused and engaged, leading to lost productivity and missed opportunities for collaboration and problem solving.

[0004] Furthermore, the lack of face-to-face interaction and non-verbal emotional cues can make it difficult for participants to build consensus and establish trust, leading to potential misunderstandings and conflicts. This highlights the need for new video conferencing methods that promote more dynamic and interactive exchanges between participants.

[0005] In response to these challenges, there is a growing demand for innovative video conferencing technologies that improve the overall meeting experience. Some of these technologies include virtual backgrounds, augmented reality overlays, interactive whiteboards, and real-time collaboration tools. These features help engage participants, promote more meaningful conversations and collaboration, and ultimately improve productivity and decision-making.

[0006] Therefore, there is a clear need to continue to invest and evolve video conferencing technology to make it more engaging, interactive and productive. By doing so, we can improve the virtual meeting experience and facilitate better communication, collaboration and problem-solving among participants in remote or hybrid work environments. Summary of the invention

[0007] Frequent online meetings may become monotonous and repetitive, reducing the motivation to participate in such meetings. This is because humans are inherently emotional beings and tend to enjoy variety, and there are many types of stimulation that can achieve this. However, current virtual environment communication technologies are mainly focused on providing sufficient technical capabilities to achieve a smooth user experience, such as allowing appropriate FPS, sound quality, or graphics quality. Therefore, there is a need to develop technological alternatives that can stimulate a variety of human emotions to increase users' vitality, attention span, motivation, and even productivity in virtual environments. Virtual environments may include indoor environments, such as offices or restaurants, as well as outdoor environments, such as parks or farms.

[0008] Solutions in the prior art provide some basic building blocks, but do not provide an all-in-one solution for generating engaging video conferences or virtual meetings. Video conferencing is already well known in the art, with applications such as Zoom and Teams. However, virtual meetings may include traditional video conferences, and may also include more complex virtual environments that simulate real-world scenes, such as restaurants or conference rooms. Certain embodiments of the present disclosure provide improved automatic response artificial intelligence logic based on dynamic information, which is customized to influence the mood of an event by processing inputs into desired outputs. Embodiments may include an artificial intelligence camera system and method based on multimodal data streams, which processes multiple inputs into outputs. These inputs may be sound or video information provided by users through their corresponding cameras and microphones, captured when users participate in virtual activities. More inputs may be captured by one or more virtual broadcast cameras in the virtual environment. Each user has a user graphical representation, or avatar, in the virtual environment, presented in the form of their own camera information. The background of the user's camera information may have been removed, so that the avatar is just a real-time cutout of the user and is sent to the virtual environment. The system includes a server storing a virtual environment and is configured to process and distribute camera feeds received from multiple participants. The system can use a WebRTC architecture that uses a scheme such as P2P, client-server, or a combination thereof, as discussed in U.S. Patent 11,218,522.

[0009] Virtual environments provide users with opportunities to meet and interact remotely. When interacting in such a virtual environment, users may need to actively adjust their camera views in order to be able to follow the action in the virtual environment. For example, in a roundtable meeting with many users, when switching from one speaker to another, the users may need to make one or more actions to adjust their camera views in order to be able to see the new speaker. Yerli's patent application No. 17 / 060,516 proposes a virtual broadcast camera that can be used to capture action in a virtual environment that the user can directly see. However, in this case, a client device is required to manage the virtual broadcast camera, which also requires manual operation.

[0010] In addition to capturing live action presented to users, footage shot by virtual cameras such as virtual broadcast cameras can also be used to create replays of real or virtual events. Typical approaches require sending video streams to storage devices in servers, allowing controllers to review and manually select and / or edit footage that may be of interest to viewers, which involves a significant amount of work. Finally, typical virtual events follow a certain predefined sequence, which can make the meetings dull and monotonous, all of which is exacerbated by the remote nature of these events, leading to fatigue and boredom.

[0011] According to certain aspects of the present disclosure, systems and methods for AI camera systems based on multimodal data streams for processing multiple inputs into outputs are provided. The inputs may be camera and audio feeds shared by a user with a virtual environment. The system includes a server storing the virtual environment and is configured to process and distribute camera feeds received from multiple participants. The system allows various possible outputs to be configured based on the received inputs. The system further uses preset or contextual scenes to allow programmatic selection and / or modification of outputs. The output may be presented in one or more surfaces of the virtual environment, for example, in a virtual screen of a virtual environment where an event is being broadcast, and may be configured to generate a more attractive virtual environment. The camera output may take the form of one or more live, unmodified camera feeds; feedback of real-time camera adjustments; or scene playback, for example, in order to produce a "best" most relevant scene including an event.

[0012] For example, if the system detects that a participant is expressing emotional cues of enthusiasm and excitement, it can adjust the mood of the event to match that sentiment. This can be done by adjusting lighting and sound effects, changing the tone and body language of the speakers, and adjusting the pace of the conversation. Similarly, if the system detects that a participant is bored or disengaged, it can adjust the mood to inject more energy and excitement into the conversation.

[0013] A user's emotional cues can be based on several factors, such as body language, facial expressions, tone of voice, and word choice. Here are some examples of emotional cues:

[0014] Body language:

[0015] Crossing your arms or legs may indicate defensiveness or discomfort.

[0016] Leaning forward can show interest or engagement.

[0017] ·Fidgeting or tapping can indicate nervousness or impatience.

[0018] Slouching or avoiding eye contact may indicate disinterest or discomfort.

[0019] An open and relaxed body posture can indicate confidence and ease.

[0020] Facial expressions:

[0021] ·Smiling can express happiness or friendliness.

[0022] · A frown or furrowed brows indicates sadness or anger.

[0023] A raised eyebrow indicates surprise or interest.

[0024] A clenched jaw or lips pressed together can indicate anger or nervousness.

[0025] Squinting or squinting your eyes indicates doubt or disbelief.

[0026] Tone:

[0027] A high-pitched or trembling voice may indicate nervousness or fear.

[0028] A monotonous voice indicates boredom or disinterest.

[0029] Loud or raised voices indicate anger or frustration.

[0030] Soft, slow voices can indicate sadness or worry.

[0031] ·Fast, energetic sounds indicate excitement or enthusiasm.

[0032] Word Choice:

[0033] Positive words such as "amazing," "fantastic," and "great" can indicate happiness or enthusiasm.

[0034] Negative words like “terrible,” “horrible,” and “disappointing” can express anger or disappointment.

[0035] Neutral words or phrases, such as "interesting," "okay," or "whatever," may indicate a lack of interest or indifference.

[0036] Words or phrases that reflect personal experiences, such as “I feel” or “It seems to me”, may indicate a personal emotional response.

[0037] The dataset can be used to train neural networks to recognize these emotional cues.

[0038] Using machine learning to detect emotions can also enhance the naturalness and spontaneity of virtual events. In face-to-face conversations, participants naturally adjust their tone, body language, and speaking patterns based on the emotions of other participants. By using machine learning to detect emotions, the system can replicate this natural behavior in virtual events. The system can analyze the emotions of participants and adjust the conversation to reflect those emotions, creating a more natural and spontaneous conversation. Embodiments can use additional inputs such as whether people are talking, whether people are using gestures and the types of those gestures, body language, or other inputs received from keyboards or other sensors, such as touch, pressure, temperature, humidity, light, heartbeat, etc. Additional information that embodiments can process is how users interact with or move in the virtual environment.

[0039] Further inputs may include so-called "dead" information, such as background audio and video information that has been cropped through a cutting process. There may be security or privacy issues that may be remedied through certain opt-in requirements or other legal means. Noise removal processes may be used to extract or enhance foreground or critical speech, and to classify and segment other sounds.

[0040] Each of these inputs can be analyzed, modeled, and processed in different ways and at different intervals to help select interventions designed to drive behavior in the virtual environment of a virtual meeting. For example, if someone has a dog barking in their background, the system’s extraction algorithm will recognize this and have a short list of criteria. The system can then add a video / animated 3D model of a dog in the scene outside the window. Or if they have a kettle on, the kettle in the room will start to produce steam. Or if it’s prayer time, a golden glow can be in the sky. Causes and effects don’t have to be direct or literal. They can be cumulative or combined. For example, if two or three users make a Teams notification at the same time, the scale of a 3D object representing the sound might change. Or if there’s a growing vacuum cleaner noise, the robot vacuum cleaner can start moving closer to the user producing the sound. Objects in the room can represent the cumulative background noise level in the room in new and artistic ways, such as water ripples on a sphere, fireworks outside, or subtle vibrations of fixtures / furniture.

[0041] A similar approach can be taken with video background information. For example, if someone places a cup on a table, it can be simulated as a real scene. If there are green curtains behind them, a decorative detail in the virtual environment can change accordingly. If it is sunny when they look out their window, the weather outside the 3D room can also change accordingly. Again, these elements can be combined. For example, if it is sunny in location A and cloudy in location B, the effect of sunlight breaking through the clouds or a rainbow can be simulated. Or there can be a shared bookshelf or art wall in the virtual environment, integrating the reading materials and background picture tastes of everyone there. In addition, if someone has a painting of a beach background, the background in the virtual environment can become a beach scene. If there is a Star Trek poster, the background can become space, and so on.

[0042] More generally, if their lighting varies between two thresholds, the virtual environment can match that behavior. If they have a piece of art in the background, this can be recognized and applied to their personal virtual environment. Or, if they are wearing a brand’s clothing or using a brand’s technology, this can be de-branded or blurred. If they have a book on their shelf, that book can be imported and linked for others to see, rather than pasting a URL into the chat. If they hold an object up to the camera, this recognition process can help remove the friction of digitizing analog information, such as using OCR to AI segmentation.

[0043] Embodiments may also use, for example, neural radiation fields to reconstruct a 3D space using one or more 2D images. For example, someone could point a phone at a chess board, and a matching 3D playable chess board could appear on the table in the virtual environment.

[0044] In virtual events such as virtual meetings or webinars, replicating the natural flow and emotions of face-to-face conversations can be challenging. However, by using machine learning to detect emotions, the system can effortlessly simulate realistic natural conversations in a simple manner. Overall, using machine learning to detect emotions can have significant benefits in increasing the dynamics, naturalness, spontaneity, excitement, and energy of virtual events. The system can analyze the emotions of participants in real time and adjust the tone and mood of the event accordingly, creating a more engaging and dynamic experience. By replicating the natural behavior of face-to-face conversations, the system can also enhance the naturalness and spontaneity of virtual events.

[0045] Embodiments include methods and systems for operating an interactive virtual conference platform storing multiple virtual scenes in one or more virtual environment systems. The platform may also store multiple contextual scenes, such as business meetings or parties. Emotional cues may also be stored by the platform. The platform may receive multiple requests to join a virtual meeting. Participants may connect to a virtual meeting by connecting sessions including video and audio streams together. The platform may include a server that uses one or more neural networks to analyze multiple video or audio streams. The analysis may automatically detect contextual scenes or one or more emotional cues in multiple contextual scenes from input data. Based on the analyzed input data, the system and method may select an intervention from an intervention database based on the analyzed input data, the detected contextual scenes, or the detected emotional cues. Next, the platform may read the intervention from the intervention database. After reading the intervention, the platform may intervene in the virtual meeting based on the intervention read from the intervention database. The intervention may include at least one change to an output audio signal or an output video signal.

[0046] In some embodiments, the input data also includes one or more of typing speed, typing volume (cancel), gestures, amount of speaking time, facial (micro)expressions, mouse / slide speed, geolocation, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, language toxicity, device, or speech cadence.

[0047] In some embodiments, the platform also includes a configuration application that allows certain options of the platform to be set. For example, the configuration application can be configured to allow the selection of interventions independent of the contextual scenarios of the participant interactions. In another example, the configuration application may be configured to allow the selection of interventions based on input data related to the contextual scenarios. In this example, the input data in a specific contextual scenario may produce different interventions depending on the contextual scenario. For example, if in a questionnaire-type scenario, the participant raises his hand, the platform may select an intervention measure, such as shining a spotlight on the participant and playing a piece of music at the same time, but if in a business scenario, the participant raises his hand, the platform may select an intervention measure, such as only turning the virtual camera to the participant, while lowering the microphone volume of all other participants, without playing music or shining a spotlight on the participant.

[0048] The platform may also receive inputs including one or more data sets corresponding to a plurality of contextual scenarios into one or more neural networks. The one or more neural networks may include one or more of a convolutional neural network (CNN) and a recurrent neural network (RNN).

[0049] The system can be configured to receive feedback about the intervention and apply the feedback to one or more neural networks to train the one or more neural networks to apply to future interventions. The feedback can include a ranking from at least one user or a physical reaction from one or more users detected via a video or audio stream.

[0050] The intervention may include a change to the virtual camera angle, the lens size, or the virtual camera motion. The intervention may also include a change to the brightness of the output video signal, a change to the hue of the output audio signal, or a change to the hue of the output video.

[0051] The intervention may also include summarizing the events occurring in the virtual environment by selecting the most interesting scenes of the events to create a summary thereof. In this embodiment, recordings may be made by one or more virtual cameras while capturing a large amount of data. These recordings may be presented on one or more surfaces of the virtual environment, such as on a screen of the virtual environment or projected on one or more of its walls.

[0052] Embodiments may also include reading selection intervention based on one or more user profiles. A user profile may include one or more user settings for contextual scenarios associated with pre-selected intervention criteria.

[0053] In further embodiments, the user profile includes data related to the user's learning style, wherein the intervention includes adjusting the presentation of data to the user based on the user's learning style to facilitate learning. In further embodiments, the platform is configured to read user feedback and adjust the data presentation based on the user feedback. In other embodiments, the platform is configured to categorize users with similar learning styles and cluster the data accordingly for presentation to users under the same categorization to facilitate group learning.

[0054] Further embodiments may include multiple client video conferencing applications and an interactive video conferencing platform configured to receive one or more video streams or audio streams from the multiple client video conferencing applications. The interactive video conferencing platform can analyze the one or more video streams or audio streams and detect contextual scenes based on the analysis. These embodiments may also include an intervention database including multiple interventions, and based on detecting the contextual scenes, these embodiments may read and implement interventions corresponding to the contextual scenes. These embodiments may also include an output signal including an output audio signal or an output video signal, wherein the interactive video conferencing platform is configured to modify the output signal based on the intervention corresponding to the contextual scene. Interventions may include changing one or more camera viewing angles and environmental changes.

[0055] Embodiments may also include various non-transitory computer-readable media, including instructions that can be executed on a processor. One instruction may be to connect multiple users to a virtual conference based on multiple requests to join the virtual conference, wherein each session includes one or more video or audio streams that together form multiple video or audio streams. A further instruction may be to analyze multiple video or audio streams to detect contextual scenes from multiple contextual scenes stored in a scene database. The instructions may also include instructions for automatically detecting contextual scenes of multiple contextual scenes stored in the scene database. After detection, the media may select an intervention from an intervention database based on the contextual scene. After selection, there may be an instruction to read an intervention based on a contextual scene in the intervention database. Then, after reading the intervention, the media may include instructions to intervene in the virtual conference based on the intervention read from the intervention database, wherein the intervention includes at least one change to an output audio signal or an output video signal.

[0056] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary aspects of the disclosure and, together with the description, explain the principles of the disclosure.

[0058] Figure 1 An overview of an example environment is presented in which systems, methods, and other aspects of the present disclosure may be implemented.

[0059] Figure 2 A general example of a process flow 200 of the present systems and methods is shown.

[0060] Figure 3 An example process for conducting an intervention in accordance with some embodiments is shown.

[0061] Figure 4 An example flow chart of an exemplary method of intervening using artificial intelligence in a computing environment according to aspects of the present disclosure is shown.

[0062] Figure 5 An example model for using artificial intelligence in a computing environment in accordance with aspects of the present disclosure is shown.

[0063] Figure 6 An example virtual broadcast environment is shown in accordance with aspects of the present disclosure.

[0064] Fig. 7A Example inputs and outputs according to aspects of the present disclosure are shown.

[0065] Figure 7BOther example inputs and outputs according to aspects of the present disclosure are shown.

[0066] Figure 8 A number of potential AI camera scenarios 800 are shown for certain embodiments of the present disclosure.

[0067] Fig. 9 An example flow chart according to certain embodiments of the present disclosure is shown.

[0068] Fig.10 An example environmental intervention process according to certain embodiments of the present disclosure is shown.

[0069] Fig.11 An example system of components capable of executing instructions for the methods of various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0070] The following embodiments describe systems and methods for generating more engaging virtual conference presentations. Current virtual conferences are relatively static and boring because they either provide a single video feed of the speaker or display boxes of several attendees at the same time. Embodiments of the present disclosure may allow for different camera effects or other interventions to produce more engaging presentations.

[0071] The subject matter of the present specification will now be described more fully below with reference to the accompanying drawings, which form a part of the present specification and illustrate specific exemplary embodiments by way of illustration. An embodiment or implementation described herein as "exemplary" should not be interpreted as being preferred or advantageous, for example, superior to other embodiments or implementations; on the contrary, it is intended to reflect or indicate that the embodiment is an "example" embodiment. The subject matter can be embodied in a variety of different forms, and therefore, the subject matter covered or claimed is intended to be interpreted as not limited to any exemplary embodiment set forth herein; the exemplary embodiments are provided for illustration only. Similarly, the subject matter claimed or covered also has a fairly wide range. Among other things, for example, the subject matter can be embodied as a method, device, component or system. Therefore, the embodiment can, for example, take the form of hardware, software, firmware or any combination thereof (except software itself). Therefore, the following detailed description is not intended to be understood as limiting.

[0072] Throughout the specification and claims, terms may have implied or implicit subtle meanings beyond those explicitly stated. Likewise, use of the phrase "in one embodiment" herein does not necessarily refer to the same embodiment, and use of the phrase "in another embodiment" herein does not necessarily refer to a different embodiment. For example, it is intended that claimed subject matter includes in whole or in part in conjunction with the example implementations.

[0073] The use of the following terms may be interpreted in the broadest reasonable manner, although it is used in conjunction with the detailed description of certain specific examples of the present disclosure. In fact, certain terms may even be emphasized below; however, any term intended to be interpreted in any limited manner will be disclosed and explicitly defined as such in this detailed description section. Both the foregoing general description and the following detailed description are merely exemplary explanations and do not limit the claimed characteristics.

[0074] In this disclosure, the word "based on" means "based at least in part on." The singular forms "a," "an," and "the" include plural references unless the context dictates otherwise. The term "example" is used herein in the sense of "example" rather than in the sense of "ideal." The word "or" is inclusive and refers to any one, any one, several, or all of the listed items. The meaning of the terms "comprise," "including," "comprising," "including," or other similar variations are intended to cover a non-exclusive inclusion, i.e., a process, method, or product that includes a list of elements does not necessarily include only those elements, but may include other elements that are not expressly listed or that are inherent to such a process, method, article, or apparatus. Relative terms, such as "substantial" and "general," are used to indicate a possible variation of ±10% of a stated or understood value.

[0075] Now referring to the accompanying drawings, Figure 1 An overview of an example environment (or system) 100 according to one or more embodiments of the present disclosure is shown. The environment 100 may include, for example, a first user device 110 and a second user device 120, which are configured to communicate with a network system 130 and a collaborative interactive video conferencing platform 140. Although two user devices 110 and 120 are shown in the environment 100, according to the present disclosure, additional user devices may be provided within the environment 100 to synchronously or asynchronously communicate with the network system 130 and / or the collaborative interactive video conferencing platform 140 and participate in collaborative content creation by multiple users.

[0076] According to various aspects of the present disclosure, the network system 130 may define one or more networks including wired or wireless networks. The network system 130 may include, for example, the Internet and / or one or more cloud networks. In addition, the network system 130 may include a public network such as the Internet, a private network such as an intranet, or a combination thereof, and may utilize various networking protocols that are now available or developed later, including but not limited to TCP / IP-based networking protocols. The network system 130 may be configured to communicatively couple the user devices 110 and 120 to the collaborative interactive video conferencing platform 140 to enable data communication between the user devices 110, 120 and the collaborative interactive video conferencing platform 140. The network system 130 may generally be enabled to use any form of computer-readable or machine-readable media to transmit information from one device to another. The network system 130 may include a communication method by which information can be transmitted between computing devices. The network system 130 may be considered a public or private network connection, and may include, for example, encryption or other security mechanisms employed in a virtual private network or on the public Internet.

[0077] In one embodiment, user devices 110 and 120 may communicate directly with collaborative interactive video conferencing platform 140, or indirectly with collaborative interactive video conferencing platform 140 via network system 130 or other available communication channels. In the case where user devices 110 and 120 communicate directly with collaborative interactive video conferencing platform 140, collaborative interactive video conferencing platform 140 may be implemented and configured to facilitate communication via one or more communication methods described, for example, with reference to network system 130 above.

[0078] According to aspects of the present disclosure, the collaborative interactive video conferencing platform 140 may include a server system 142, a video storage system 144, a neural network 146, a virtual environment system 148, and an intervention database 150. In some embodiments, according to aspects of the present disclosure, the interactive video conferencing platform 140 may be one or more servers that may be configured to perform some or all of the functions of the server system 142, the video storage system 144, the neural network 146, the virtual environment system 148, and the intervention database 150. In the present disclosure, one or more systems may include various implementations that may include various electronic and computer systems. One or more implementations described herein may implement functionality using two or more specific interconnected hardware modules or devices with associated control and data signals that may be communicated between and through the modules, or as part of a dedicated integrated circuit. Therefore, the system includes software, firmware, and hardware implementations.

[0079] According to aspects of the present disclosure, the server system 142 may include one or more data servers or databases, which may be located on a cloud server / network, an edge server / network, in the network system 130, and / or a location where the server system 142 may be directly or indirectly integrated with the collaborative interactive virtual conference platform 140. According to embodiments of the present disclosure, the server system 142 may store and process data received from the user devices 110 and 120 and / or the network system 130 to produce a participating virtual conference. In addition, the server system 142 may receive data or command signals from the video storage system 144, the neural network 146, the virtual environment system 148, and the intervention database 150, and send data or command signals to the video storage system 144, the neural network 146, the virtual environment system 148, and the intervention database 150 to facilitate the execution of the participating virtual conference.

[0080] According to aspects of the present disclosure, the video storage system 144 can receive one or more inputs or commands from the first user 112 and the second user 122 through their corresponding user devices 110 / 120. As described above, the environment 100 can include more than two users or user devices 110 and 120. Therefore, the video storage system 144 can receive inputs and commands from more than two users or user devices. Alternatively, the video storage system 144 can receive one or more inputs or commands from only the first user 112 or the second user 112 depending on the type of collaborative session. That is, the collaborative interactive video conferencing platform 140 can facilitate one or more virtual meetings.

[0081] In one embodiment, the first user 112 and / or the second user 122 may input commands to the first user device 110 and the second user device 120 synchronously or asynchronously, respectively. The video storage system 144 may receive input commands from the user devices 110 and 120 directly or indirectly through the network system 130 and / or the server system 142. The video storage system 144 may then transcribe or convert the input commands into text data. For input commands that may be received in the form of text data, it may not be necessary to convert them into text data by the video storage system 144. The video storage system 144 may then analyze the converted text data or the received text data by capturing the semantics of the text data (e.g., understanding the context of the conversation), detecting repetitions and / or redundancies (e.g., unnecessary information), marking the text data, and / or detecting prompts, according to the type or form of the input command. If necessary, the video storage system 144 may clean or modify the analyzed text data by eliminating (e.g., detected repetitions and / or redundancies). The video storage system 144 may then continue to extract prompts (if any) from the analyzed and cleaned text data. In some embodiments, the video storage system 144 can utilize a trained machine learning model. The video storage system 144 can then send the extracted, clean prompts to the virtual environment system 148. In some embodiments, data conversion or transcription can occur on the user device 110, 120. Therefore, the user device 110, 120 can implement computer code that converts audio or other types of data into text and sends the transcribed text to the collaborative interactive video conferencing platform 140. Alternatively, as described above, the data transcription or conversion can be performed in the collaborative interactive video conferencing platform 140 in the server system 142 so that the input device 110, 120 does not need to perform any data transcription or conversion. Alternatively, all functions of the server system 142 can be performed by the neural network 146, including text transcription or conversion, input or command data analysis, and / or input or command data cleaning.

[0082] In one embodiment, the video storage system 144 can perform a labeling process on any type of input or command data, such as image data, video data, and / or other types of audio data. For data that does not include text data, the labeling process can be viewed as a way to transcribe commands into text. For example, if the tone of voice of users 112, 122 is associated with a particular emotional prompt by the video storage system 144, the video storage system 144 can generate labels such as anger, happiness, sadness, worry, anger, etc., which can be determined by one or more neural networks 146. Multiple participants may exhibit similar, different, or even contradictory emotions. The video storage system can determine the emotions of each participant in real time based on a set of training data.

[0083] According to aspects of the present disclosure, the virtual environment system 148 may analyze certain factors, such as the voice intonation, speech, and movement of participants, and make certain interventions in the virtual meeting to keep the virtual meeting moving smoothly or to achieve some other goal defined by one or more users.

[0084] According to aspects of the present disclosure, the virtual environment system 148 can facilitate one or more computing environments that users 112, 122 in the generation environment 100 can use to collaborate to create content. The computing environment may include, for example, an augmented reality environment, a virtual reality environment, and a two-dimensional (2-D) or three-dimensional (3-D) simulation environment, but is not limited thereto. Examples of computing environments and user collaboration in computing embodiments are also disclosed in co-pending U.S. patent application Ser. No. 17 / 006,327, which is incorporated herein by reference in its entirety. In addition, any machine learning algorithm disclosed in the co-pending application can be incorporated for use alone or in combination with the machine learning model used by the video storage system 144 and / or the virtual environment system 148.

[0085] The system may allow the user to configure various possible outputs based on the input received. The system may further use presets, or contextual scenarios, to allow the program to select outputs and / or modify outputs. Different scenarios may include a business meeting in a small office or large conference room, a family birthday party, a doctor's office, or a nightclub.

[0086] Thus, the system utilizes multiple data points from many data sources, including user and non-user data sources. User data sources include data generated by user avatars, such as their graphic data, the position and orientation of the avatar, their voices, actions, activities performed by the user avatar, etc. Non-user data includes objects in the virtual environment, ambient noise, ambient music, lighting, brightness, color, etc. The data can be processed to create camera outputs that represent further stimulation of the user to produce emotions and behaviors. The processing of the data includes a logical interpretation of the input by the neural network implemented by the system, which can be used to create corresponding camera outputs. The processing of the data includes a logical interpretation of the input by the neural network implemented by the system, which can be used to create corresponding camera outputs. The generation of further stimulation can help access a wider range of human emotions of users interacting in the virtual environment, which can enhance the user's motivation, empathy, learning, connection and overall energy, create more interesting and playful virtual events, and potentially increase attention span, learning ability and productivity. The systems and methods of the present disclosure can be used for any type of virtual event hosted in a virtual environment, such as virtual work meetings, video games, sales meetings, virtual conferences, seminars, etc.

[0087] The system may implement machine learning-based empathy capabilities to create intelligent and compassionate virtual environments and generate interventions, such as changing one or more camera perspectives or environmental changes. In the current disclosure, the term "empathy" refers to the ability of a computer system that employs artificial intelligence algorithms to logically understand a user's psychological perspective, emotions, and behavior, and to use this logical understanding to create outputs that influence the user.

[0088] Empathy as employed in this disclosure can be divided into 3 categories:

[0089] - Cognitive empathy: The ability to understand users’ thoughts, intentions, and motivations to influence their behavior through one or more interventions.

[0090] -Emotional empathy: The ability to understand emotions and moods. This can be done based on subtle non-verbal messages such as gestures, tone of voice, speaking or typing speed, pauses between words and / or sentences, facial expressions, hand gestures, etc.

[0091] - Social Empathy: The ability to cognitively and / or emotionally understand a group to influence their behavior through one or more interventions. The system can read different cognitive and emotional cues from several types of inputs and determine the dominant perspectives, emotions, and behaviors of the group to define interventions that can create desired outputs. The system can also categorize and separate interventions based on their socially determined cognitive and emotional strengths, for example, by providing a certain type of camera output to a group that primarily feels one way and another camera output to another group.

[0092] Embodiments may use empathy detection to generate interventions to create a positive virtual meeting experience.

[0093] Example interventions may be, for example, raising the volume of a user and lowering the volume of other users, highlighting a user, modifying the weather of the virtual environment, changing the lighting and / or color of the virtual environment, generating a pop-up window with text prompting the user to take an action, generating audio or haptic data prompting the user to take an action, modifying the viewpoint of a virtual camera, one or more user avatars, etc. The resulting user behavior is further captured by the system and may be used as new input to generate corresponding new outputs.

[0094] Camera output can be one or more of the following:

[0095] - Indirect interventions: These are changes in color, sound, ambiance, etc. that do not directly tell the user what behavior change is expected of them. For example, changing the background color, highlighting someone, changing the music or weather are all indirect interventions.

[0096] - Direct intervention: Text, visual or audio output that is directly addressed to the user and is intended to get the user to take some action, such as asking this question, providing additional data, providing a summary of information, analyzing information, etc. This is associated with explicit feedback, rather than something that the user needs to interpret, such as being associated with explicit rules, such as if a user has not spoken for a long time, a text prompt can be displayed to ask the user a question. In another example, a salesperson is in a sales meeting with a customer and the customer has not spoken for a long time. A time rule is configured in the system, and when it measures that the customer has been silent for a long time, the system directly intervenes with the salesperson to remind him to ask a question. Such a time rule can be configured to send a prompt after a preset number of seconds or minutes, and the time rule can also be affected by the semantic context of the conversation or other factors. For example, if the previous conversation contained an instruction from one user to another user to give them 5 minutes to review a document, the time rule will take that time into account while not causing any interruption. The prompt can be in the form of a text, tactile or audio prompt. Text is the default, but other ways are possible; or

[0097] - Massive behavioral interventions: These interventions can be applied to video games or other applications that require group interaction. In the case of video games, a mass behavioral intervention could prompt the player to change the scene, or have the world be hit by a meteorite and have the player go somewhere else, or have a monster appear and have the player go somewhere else. These interventions could be applied to, for example, meetings, where if the atmosphere is tense, a mass intervention could prompt everyone to take a break. This intervention does not provide any other choice than to perform the prompted behavior.

[0098] Figure 2 A general example of a process flow 200 of the present system and method is shown. The system may have a plurality of inputs 205 corresponding to system inputs 290-299. System inputs 290-299 may be, for example, video or audio streams, typing speed, typing volume (cancellation), gestures, amount of speaking time, facial (micro) expressions, mouse / slide speed, geographic location, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, language toxicity, device or voice cadence, bandwidth, throughput, and strength of input signals.

[0099] The system may use multiple neural network rhythms 210 (e.g., facial recognition, voice analysis, etc.) corresponding to neural networks 270-285 that process one or more input categories (e.g., sound 295, video 299, system input 290, etc.) and produce different forms of network data 215 corresponding to network outputs 265 and 260, depending on the configuration of the system and the system inputs 290-299 received. The neural networks 270-285 can operate at different neural network rhythms 210. For example, parallel input streams can be processed at different intervals or beat rates depending on what they capture. Emotions may appear every 5-10 seconds. Movements may be processed several times per second. This may also vary depending on the focus of the conversation. For example, a speaker may be processed more frequently than someone who is not participating in the conversation. The way each parallel input stream is processed may vary depending on the identity of the input person(s), what they are doing, and other factors. These network outputs 260 and 265 are further processed by executing rule-based operations or engine code 220 stored in the form of logic tests 255 to produce one or more corresponding camera outputs. The camera output may be presented in one or more surfaces of the virtual environment, for example, in a virtual screen of the virtual environment where the event is being broadcast. The camera output may take the form of one or more environment changes 225, corresponding to environment changes 235 and 240:

[0100] -Live, unmodified webcam;

[0101] - Camera field of view adjustment in near real time; or

[0102] - Scene playback, e.g. in order to generate the "best" most relevant scene including an event.

[0103] The system may then implement environment changes 235 and 240 via state change 230 , corresponding to elements 250 and 245 , respectively.

[0104] Embodiments may include a processor configured to receive data from multiple sources, such as cameras, microphones, sensors, and other appropriate input devices. The processor may then pre-process the data by applying filters and converting the data into a format suitable for analysis. In some embodiments, the data may be segmented into smaller portions or frames to improve the accuracy of the analysis.

[0105] The preprocessed data can then be passed through a convolutional neural network (CNN), a type of deep learning algorithm that is particularly well suited for image analysis. CNNs can be trained using large datasets of labeled images corresponding to various emotional states. During training, the CNN learns to recognize patterns in the data that are associated with specific emotions. The trained CNN can then classify new data based on the patterns it has learned.

[0106] In some embodiments, the system can use multiple CNNs, each trained on a specific type of input. For example, one CNN can be trained on facial expressions, while another can be trained on voice intonation. The outputs of these CNNs can then be combined using a fusion algorithm to produce a more accurate prediction of the emotional state.

[0107] In addition to CNN, other AI algorithms, such as recurrent neural networks (RNN) or support vector machines (SVM), can be used depending on the type of input and desired output. For example, RNN can be used for time series data such as speech intonation, while SVM can be used for gesture recognition.

[0108] In general, the present disclosure provides a method and system for using machine learning to detect emotions using various inputs. The use of CNN and other suitable artificial intelligence algorithms allows accurate and robust detection of emotional states, which may be useful in various applications such as market research, mental health diagnosis, and human-computer interaction.

[0109] The camera output displayed in the camera playback can be configured to prompt one or more behavioral modifications of one or more users in the virtual environment to affect the mood of the event and its dynamics, thereby creating a more vivid, natural and realistic user experience. The camera adjustment output may include, for example, zooming in or out of the user, panning the camera, changing the angle of the camera, adding camera effects, modifying the camera's field of view, etc. The resulting user behavior is further captured by the virtual camera and can be used as new input to generate corresponding new output.

[0110] Audio input can be captured by the user's microphone, including the user's voice data, music in the virtual environment, typing sounds, the pitch and speed of voice, the rhythm and prosody of speaking, etc. The video input can be captured by real and virtual cameras, and can include virtual slices of various users and elements in the virtual environment and their positions and directions, the current viewing angle of each user's virtual camera, the user's micro-facial expressions, gestures, head movements; and system input can be captured by the processing system of the virtual environment, including typing speed, loading speed, frames per second (FPS), mouse / sliding speed, current system load, system capacity (including available bandwidth, available computing power, available memory), etc.

[0111] Neural networks can use suitable artificial intelligence algorithms that take in data, such as visual data, and pass it through a logical graph that applies weights and returns a numerical response for certain attributes. Depending on the type of input and desired output, a combination of some AI algorithms can be used. An example of one of these artificial intelligence algorithms is a convolutional neural network.

[0112] The network outputs may be one or more categories of processed inputs that are further processed by executing rule-based operations stored in the form of logical tests to produce one or more corresponding camera outputs.

[0113] The camera output, also known as intervention, can be configured to prompt one or more behavioral modifications to one or more users in the virtual environment to have an impact on the mood of the event and its dynamics, which can create a more lively, spontaneous and realistic user experience.

[0114] The camera output may be personalized in a manner that varies from user to user based on a user profile. The user profile may include data such as the user's background, preferences (e.g., topics of interest, music, food, etc.), culture, personal history, age, social class, personality, learning type, etc. Preferences may also include pre-selected intervention criteria, such as whether the user prefers large or small interventions, choices between specific interventions based on various contextual scenarios or emotional cues. The system may generate further user profile data that is used to continuously update the user profile based on their interactions in the virtual environment. The system continuously learns from each user and uses this data to further improve itself and better understand each user, thereby improving its empathy. Thus, multiple "optimal" replays may be generated based on the user's profile, with at least one possible replay for each user.

[0115] In addition to replays, embodiments may generate an automatic summary of the virtual meeting based on inputs such as detected emotional cues, body language, etc. The summary may be an intelligent analysis indicating one or more meeting outcomes, such as whether the meeting was happy, sad, angry, bad, or resulted in sales. The automatic summary may also include statistics such as whether there were a lot of action points, the amount of laughter, the number of questions asked, or the number of diverse people participating in the virtual meeting, etc. Embodiments may connect directly to social media to automatically report the automatic summary to the social media platform.

[0116] Indirect environmental interventions can be presented to users at an individual or group level, so depending on the desired outcome, the system can determine that a certain intervention should be shown to some users but not others.

[0117] In an example of an indirect intervention, if a user has not spoken in a meeting for a while, the intervention could be to play a personalized piece of music, which is a song that the user likes and is played only to that user, to prompt him or her to perk up and participate in the conversation. In another example, if the system determines that the atmosphere of the virtual meeting is one of anger, for example because most users show signs of anger (e.g., facial expressions, tone of voice, gestures, words used, etc.), then the system may decide to add more cool colors, such as blue, green, or purple, to the virtual environment and / or play soothing background music in the virtual environment to change the atmosphere of the virtual meeting. Songs and / or colors can be selected individually based on the user's profile, so even if similar interventions (e.g., modifying colors and music) are selected, each user may end up seeing and hearing something different. The system may also decide to modify the user's current virtual camera's perspective and / or show some content in front of the virtual camera that may affect his or her mood.

[0118] Likewise, direct intervention messages may be sent to many users simultaneously, with each user's message being different depending on their user profile.

[0119] The same logic applies to large-scale behavioral interventions. For example, if a game level is to be changed, the system may display a message to all users asking them to go to a certain area. The text color of this message will be different depending on the user's profile. In addition, the music heard by each user may also be different depending on their profile.

[0120] In one implementation, the virtual environment can be a 3D board game. Interventions can be large-scale, such as exploding the board. A variety of different interventions can be made to the game based on detected inputs such as behaviors, actions, emotions, etc. This can have a huge impact on product engagement, retention, and churn.

[0121] In another embodiment, such as a training, sales or educational environment, combining a 3D environment with a video conferencing service can be very useful. Normally, it is very expensive to produce exciting graphics and light shows. In a virtual environment, they are much cheaper and can produce a similar effect.

[0122] exist Figure 3 In the example of FIG. 3 , by accepting several types of input 300 to determine existing behavior 305, processing the data (via a neural network), and analyzing and evaluating the network output 310 via logical testing, the system can design interventions 315 in the form of camera outputs designed to prompt different behaviors in the virtual environment 320. Interventions can be determined based on learned empathy, as described herein.

[0123] Figure 4An example flow chart 400 is illustrated in which system input 405 captured by a microphone 475 and a webcam 480 is sent to four neural networks 410 for processing, each of which is configured to process different elements, such as head movement 455, hand movement 460, emotion 465, and gesture 470, including their position and orientation in a virtual environment. The neural networks 455, 460, 465, and 470 can be polled for input data regularly (e.g., every second) and can produce network data 415, such as head position 445 and voice tone 450. Polling may be performed at different neural network rhythms depending on several factors, such as who is speaking, the need for frequent sampling, bandwidth and other computing resources. Changes in rhythm can occur in real time. Selected network data 415 can be used in logic tests 440 to deduce that emotions, such as anger, can prompt corresponding outputs 425 in the form of an increase in FOV 430, a change in camera angle, moving the camera away from or toward the user, changes in weather, sound, color, etc. For example, if the voice pitch 450 rises and the head position 445 (and perhaps facial gestures) are interpreted by the system as being somehow "aggressive," then the logic test 440 might conclude, based on the training results of the neural network 410, that the user is expressing anger.

[0124] Outputs can be presented in various forms, such as weather changes in the virtual environment, such as thunder starting to sound, playing thunder sounds, amplifying the user's voice, changing the background color (for example, turning red) or fading from red to purple or blue. Thus, environmental interventions can induce certain behaviors or emotions. The behavior is further perceived by the virtual environment and further outputs are generated as needed.

[0125] In some embodiments, the system enables a record of an event in a virtual environment to be presented to the user when needed. In other embodiments, the system also enables the application of an AI algorithm that enables summarizing an event to select the most interesting scenes of the event to create a summary thereof. In the case of capturing a wide range of data including user and non-user data, recording can be performed by one or more virtual broadcast cameras. The event record can be presented on one or more surfaces of the virtual environment, or can be stored in a memory for later transmission to a user device for later playback by the user. If necessary, the event record can be further processed before being sent to the user. The event record can be detected by, for example, identifying a strong reaction of a participant. Reactions can be stored using multiple camera angles simultaneously and stored in a memory for later playback, such as after the virtual meeting ends, at the end, or at any time when a user requests playback. The playback can be on the full user screen or on a surface in the virtual environment, such as a virtual whiteboard.

[0126] Replaying the most important parts may also be based on context. For example, a business meeting may require a selection of key points, which can be detected by a combination of speech recognition and empathy-trained neural networks. Alternatively, a meeting with entertainment material will have different contexts, perhaps determining which activities elicit the strongest reactions or laughter.

[0127] Figure 5 An example model 500 is shown where the system acquires system input 505, such as sound input 590, which includes position input 592 of a person making sound in a virtual environment. The sound and position are processed 510 and combined 588 to extract different sound parameters 515, such as determining who is speaking 582, what their tone of voice is 584, the length of their speech 586, and the recipient of their message 587. For each sound parameter 515, there can be a corresponding response 520, in this example: Alex is speaking 565, his tone is calm, he has been speaking for 10 seconds, and he is talking to Bill. For such sound analysis, a spectrogram can be used to provide valuable data that can be processed to extract the desired response. The spectrogram analysis can focus on the energy, amplitude, or frequency of someone's voice, and then analyze and try to estimate the size of the curve or speed of the camera changes 540 and 545 to predict a pattern to mobilize the camera accordingly. This data can be passed through engine code 525 such as rule-based logic testing 560 to determine one or more camera changes 530, such as moving the camera 550 or modifying the cutting speed and duration of camera shots 555. For example, a rule can be that after the same person speaks for 10 seconds, there should be a camera change 540 (such as a camera cut) or another camera change 545 (such as a slow pan toward another speaker) to prompt a change in speaker, for example, which prompts Bill to interrupt Alex to ask a question.

[0128] The system can be trained so that it uses its own feedback data or user preferences or input to improve machine learning based on the user experience. For example, if the system believes after one or more rounds that a cut is performed too quickly after 10 seconds of speaking, the system can add more seconds to the next round. The system can also further perform semantic analysis and processing to determine what is happening in the event and generate camera output accordingly. For example, if the user says "We are about to end the meeting", the composition, lighting, cut rate, etc. of the camera output can be modified to create a more dramatic effect or environmental change, such as adding epic music and / or reducing the brightness of the lights in the virtual environment. The system can also further use direct feedback from users to improve itself. Different algorithms can be trained to summarize the data, such as the algorithms used by programs such as Blinkist and Newsleo, which can also be integrated into the system.

[0129] Embodiments may collect various types of feedback data before, during, and after a virtual meeting. This data may then be used to train a neural network to detect what people like or dislike and change behavior based on this information. For example, feedback may be based on a slider on a user interface (UI) that indicates that the user is enjoying the virtual meeting. The system may prompt the user to move the slider at certain times, such as when an intervention is taken, or every 5 minutes, or a combination of both. In this way, the system may be able to determine the difference between successful and unsuccessful interventions based on the user ranking corresponding to each user's slider position. Alternatively, the user may also be asked to provide a value, such as a number 1-5, or "good", "OK" or "bad". In addition, the system is able to determine the subtlety of various interventions. Sometimes a minor intervention is needed, such as a pause. Sometimes a larger intervention may be needed, such as an argument or a fight, and the meeting needs to calm down. Choosing which intervention to use will be discussed further below.

[0130] The system can also perform further processing on the spectrogram data, such as pre-smoothing, to predict where someone is, such as how they might move their lips next. In this context, for example, if someone is about to make a statement that lasts for 5 seconds, this could be set to the camera and move accordingly, or make environmental changes to the virtual environment. In this way, the system can capture the emotional energy of a meeting or event and adjust the camera or modify the virtual environment accordingly based on this data.

[0131] Pre-smoothing can be used for 2D motion information, such as pre-moving puzzle pieces, but it can do more than that, for example if a person's fingers are being tracked, that will generate a lot of data, so if it can be interpolated and smoothed, it may increase FPS. Embodiments can increase FPS by reducing the granularity of secondary features (such as fingers) to achieve higher throughput and less bandwidth consumption. There are many benefits to doing this, especially in large-scale situations, as there may be hundreds or even thousands of virtual meetings going on simultaneously. Saving a little data in each active virtual meeting may save a lot of computer and network resources.

[0132] Figure 6An overall system 600 is shown, in which multiple virtual broadcast cameras 670 are installed in a virtual environment to capture various angles that can be used to capture a greater number of details from a scene, and multiple microphones capture sound from within the scene, which can be input to prompt corresponding processing 635 and output. Various camera feeds can also be used to generate more possible data for processing 635 to generate more possible replays, define live camera adjustments, or make environmental changes to prompt behavioral changes. The system also captures the spatial position or location 665, orientation, voice content 660 and state of each object and avatar in a virtual environment, such as a virtual meeting, such as focusing on the head 640, hands 645, emotions 650 and postures 655, all of which can be reconstructed 605 from any angle to generate corresponding replays. The spatial position, orientation and state can be metadata of the input data, so it can be used to further characterize the input data to derive different types of processing 635 and outputs. The resulting camera output can be, for example, various types of camera framing 620, camera motion 625, shearing 630, environmental changes 680, etc. The system also captures speech content which can be used for semantic analysis 615 which can provide further feedback for generating result output.

[0133] The present disclosure provides a system and method for capturing video using different camera angles to enhance the visual storytelling experience. In particular, the present disclosure describes various camera angles that can be used to capture video in a unique and compelling way, including eye level, Dutch, low angle, high angle, top angle, and over the shoulder.

[0134] The eye-level camera angle is when the camera is at the same level as the subject's eyes. This angle is often used in interviews and conversation scenes to create a sense of intimacy and connection between the subject and the audience. By using an eye-level camera angle, the viewer can feel like they are in the same virtual environment as the subject and are having a conversation with them.

[0135] The Dutch camera angle, also known as the canted angle, refers to tilting the camera to create a tilted or diagonal composition. This angle can be used to create a sense of tension, uneasiness, or disorientation in the viewer. The Dutch angle is often used in horror or thriller films to create a sense of uneasiness or disorientation in the viewer.

[0136] A low-angle camera angle is when the camera is positioned below the subject's eye level, looking upward. This angle can be used to create a sense of power, dominance, or heroism in the subject. Low-angle shots are often used in action films to make the hero appear larger than life.

[0137] A high-angle camera angle is when the camera is positioned above the subject's eye level, looking downward. This angle can be used to create a sense of vulnerability or powerlessness in the subject. High-angle shots are often used in horror films to create a sense of vulnerability in the victim.

[0138] A top-angle camera angle is when the camera is placed directly above the subject, looking straight down. This angle can be used to provide a unique and interesting perspective. Top-angle shots are often used in documentaries or nature films to show the subject from a different angle.

[0139] An over-the-shoulder camera angle is when the camera is behind one character, looking over their shoulder at another character. This angle can be used to create a sense of intimacy or tension between characters. Over-the-shoulder shots are often used during dialogue scenes to show the reactions of both characters.

[0140] In general, the present disclosure provides a system and method for capturing video using different camera angles to enhance the visual storytelling experience. By using a variety of camera angles, filmmakers can create unique and eye-catching visuals that capture the audience's attention and help tell the story in a more engaging and effective way.

[0141] An upward shot is when the camera is placed below the subject, looking upward. This shot can be used to create a sense of power or dominance in the subject. In action films, upward shots are often used to make the hero appear larger than life.

[0142] A downward camera shot is when the camera is placed above the subject, looking down. This shot can be used to create a sense of vulnerability or powerlessness in the subject. Real shots are often used in horror films to create a sense of vulnerability in the victim.

[0143] An over-the-shoulder camera shot is when the camera is placed behind one character, looking over their shoulder at another character. This shot can be used to create a sense of intimacy or tension between characters. Over-the-shoulder shots are often used during dialogue scenes to show the reactions of both characters.

[0144] A close-up shot is when the camera is placed very close to the subject, capturing a detailed view of the subject's face or body. This shot can be used to create a sense of intimacy or to emphasize the subject's emotions.

[0145] An extreme close-up is when you place the camera very close to your subject, capturing small details like the eyes or lips. This shot can be used to emphasize a specific emotion or feature of your subject.

[0146] A medium shot is when you place the camera at a medium distance from your subject, capturing the upper body or waist up. This shot is often used to show the subject's body language or physical interactions.

[0147] A telephoto camera lens is when the camera is placed farther away from the subject, capturing both the full body and surroundings. This lens is often used to define a location or setting.

[0148] A single-lens camera shot is one in which the camera is positioned to capture only one subject.

[0149] A dual camera lens is where the camera captures two subjects in one shot.

[0150] A triple camera lens is one where the camera captures three subjects in one shot.

[0151] Point of view (POV) refers to the position of the camera that films a scene from the character's perspective. This lens can be used to provide the audience with a unique and immersive experience, as if they were experiencing the scene themselves.

[0152] The present disclosure provides a system and method for capturing video using different camera lenses to enhance the visual storytelling experience. In particular, the present disclosure describes various camera lenses, including top, bottom, over the shoulder, close-up, extreme, medium, long, extreme, single, dual, triple, and point of view (POV), which can be used to capture video in a unique, eye-catching, engaging, and convincing way.

[0153] Fig. 7A and Figure 7BDifferent various parameters 700 are shown, including inputs 705, 710 and outputs 715, 720 that can be combined accordingly. Preset options 710 are contextual scenarios where inputs are captured, such as the type of meeting for which the virtual meeting is intended, and outputs 715, 720 are generated, and the outputs 715, 720 can affect the type of logical tests performed on the inputs. Inputs 705 can include typing speed, typing volume (cancellation), gestures, amount of speaking time, facial (micro) expressions, mouse / slide speed, geographic location, browser, load time, FPS / tab focus, meeting title, number of participants, head position, language toxicity, device or voice rhythm, etc. The preset options 710 may be selected in advance by the meeting administrator and may further include event categories, such as whether it is a business meeting, sales meeting, classroom, talk show, Alfred Hitchcock, talk show, YouTube, slow motion, jump cut, selfie cam, predictive view, Stanley Kubrick, free mode, TikTok, fast editing rate, fade to black, J / .L editing, selfie cam, seminar or stand-up meeting, etc. For example, an input determined to be a happy expression may result in a different output 715, 720 in a workshop than in a conversation with friends, where the playback, music or environment may be set to be more formal than in a friendly conversation. The outputs 715, 720 may include lighting / brightening, depth of field (DOF), bloom (shader effect), editing style, different color settings, camera angles, particles, shadows, editing speed, 3D lookup table, camera movement and camera lens size.

[0154] For example, a conference room is different from a showroom, which is different from a sales store. Each virtual environment is different, and the input means something different in each context and can create another output 715, 720, so it's like a framework that does affect the input and output. The preset can be selected before the meeting and can be modified by the host. It's a predefined setting for a specific time period, then maybe events change dynamically, or maybe the host intervenes and changes the setting, or something happens that changes the setting so much that it becomes a new context (like an output).

[0155] Figure 8A number of potential AI camera scenarios 800 for certain embodiments are shown. These may include situations 805, such as someone speaking, a conversation between people, someone not speaking for a long time, someone speaking for too long, the meeting is about to end, someone showing anger, etc. Data inputs 810 may include 1) speech, rhythm, position; 2) speech, rhythm, position of multiple people; 3) length of time since the last speech; 4) length of speech; 5) during the meeting, no more speakers are speaking; 6) speech, rhythm, gestures, and intonation, etc. Outputs 815 may include moving the camera to the speaker or zooming, switching multiple cameras and zooming, focusing the camera on the speaker and zooming, moving the camera from the speaker to another speaker, moving the camera from the speaker to another speaker, zooming and creating dramatic camera effects. Behaviors may include focusing on the person who is speaking, maintaining attention on the person who is talking, the user starting to speak, the user finishing speaking, or the user realizing the reaction and adjusting the behavior, etc.

[0156] Fig. 9A flowchart 900 according to one embodiment is shown. The embodiment may be performed by an interactive virtual conference platform in one or more virtual environment systems and includes multiple virtual environments stored in memory. The platform may also store multiple contextual scenes and emotional clues, as shown in step 902. In this embodiment, the virtual meeting starts in step 904. This may include receiving one or more requests to join the virtual meeting from one or more user devices. The system may then associate these requests together based on a unique identifier or a pre-programmed set of values. These requests can be connected to form a virtual meeting. In step 906, the system may analyze the received input data, including video and audio feedback from one or more user devices, for pre-programmed contextual scenes or emotional clues, as further discussed above. In step 908, when the system detects such contextual scenes or emotional clues by analyzing the input data, the system may select one or more interventions from an intervention database associated with the contextual scenes and input data. Such selection may be affected by the empathy level learned by the system based on training data and user feedback. As described above, several inputs can be used to select intervention measures. For example, voice tone, user preferences, and sound level. For example, all of this information may be used to determine whether the conversation has become angry and intervention is needed to ease the conversation. The system may select an intervention from an intervention database or memory, which may be read and applied in step 910. For example, a clown may run across the screen, or cheerful music may begin playing. In step 912, the system may collect feedback to determine the success of the intervention. Success may be determined in a variety of ways. For example, if the goal is to moderate the conversation, the system may monitor the tone of voice or the words used to determine if the conversation was moderated. Or, for example, if the system is set to encourage arguing for entertainment purposes, success may depend on whether there is more anger in the conversation. Further feedback from the participants may be solicited through one or more user inputs. The user's input may be a range, such as 1-10, and may include whether they liked the intervention, how effective the intervention was for a given goal, and whether a different intervention would be preferred. The system may then collect this feedback and incorporate it into future virtual meetings to make them more successful. Finally, in step 914, the system may end the virtual meeting or repeat the process starting with step 906 until the virtual meeting ends.

[0157] Fig.10An example of an environmental intervention process 1000 is shown. For example, if people are trapped in a virtual environment 1005, the virtual environment can assess whether people are trapped by lack of verbal input or bland body language 1010. The virtual environment generates an intervention, such as generating noise / objects / characters, which causes a change in group behavior 1015. This behavior can be further measured and used as input 1020 to determine whether the intervention was successful or failed, so that future interventions can be improved or changed. For example, volume, color, tone of voice, shape, etc. may all trigger different behavioral response systems, and the neural network will be able to evaluate these behavioral systems based on the physical and verbal response of the intervention.

[0158] Fig.11 An example of a computing device 1100 of a computer system is shown. The computing device 1100 may include one or more processors 1110 (e.g., CPU, GPU or other processing unit), a memory 1120 and one or more communication interfaces 1140 (e.g., a network interface) to communicate with other devices and receive inputs from these devices, such as typing speed, typing volume (cancellation), gestures, speaking time, facial (micro) expressions, mouse / slide speed, geographic location, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, language toxicity, device or speaking rhythm, wherein the interactive video conferencing platform is configured to select the intervention based on one or more of the inputs. The memory 1120 may include volatile memory such as RAM and / or non-volatile storage such as ROM and storage media. Examples of storage media include solid-state storage media (e.g., solid-state drive and / or removable flash memory), optical storage media (e.g., optical disk) and / or magnetic storage media (e.g., hard drive). The above instructions (e.g., software or computer-readable code) may be stored in any volatile and / or non-volatile memory components of the memory 1120. In some embodiments, the computing device 1100 may also include an input device 1150 (e.g., a keyboard, a mouse, a joystick, a controller, or a touch screen) and an output device 1160 (e.g., a display, a head-up display, an AR display, a VR display, a printer). For example, if the user devices 110, 120 can be embodied as tablet computers, the user devices 110, 120 can have a touch screen and a display. The above elements of the computing device 1100 can be connected to each other via a bus 1130, which represents one or more buses. In some embodiments, the processor 1110 of the computing device 1100 includes both a CPU and a GPU.

[0159] As an example, the system can detect that one person is quiet when too many people are talking. The system can capture the noise from multiple inputs and conclude that many are active and one is passive (or silent), prompting all active users to reduce their volume and focus the spotlight on the silent user. As a result of this behavioral intervention, other users may speak less and the silent person may speak more, restoring balance to the meeting. Users are able to set in their profile how aggressively they want the system to use this intervention for themselves or others.

[0160] The face detector AI algorithm can be used to generate facial geometry coordinates. The facial geometry coordinates can create a steering vector for the orientation of the user's head. When the user turns their head, there may be two manifestations.

[0161] First, a position vector can be detected by a virtual camera facing the user in 3D space, which can move inversely to maintain the illusion that the 2D camera feed is correctly oriented within the 3D geometry of the virtual meeting, which may be in the virtual environment. This technique may be called match-moving. The user's head as a controller can also be applied to entities in the virtual environment, such as lights, casting dynamic shadows as the user moves.

[0162] Second, the position vector may be detected by a camera representing the user's first or second person view in 3D space. There may be a scaling / limiting and minimum fence logic as the user moves their head left to right, up and down, or forward and backward. This may allow the 3D engine to draw non-text geometry. However in VR 3D space the user's movement may be captured 1:1, and on the 2D screen the user's movement is scaled accordingly.

[0163] Hand gestures can be used to map the coordinates of the user's fingers in 3D space. The 3D coordinates of each joint / finger can be captured essentially in real time. This information can then be passed to a manipulated 3D model of the hand. The user's skin color can additionally be captured from representative patches on the forehead, identified by a neural rhythm detection network. This color / texture can then be mapped onto the emissive properties of the texture of the 3D model of the user's hand. This could potentially allow virtual environments to display personalized and realistic 3D hands. The hands could also be used to gesture / grasp / pinch or other movements. More movements of the hand could be mapped to any visual properties of the virtual environment, such as weather, lighting, outdoor wind, etc.

[0164] Other issues that some instances may address include standardized teaching environments that can slow down learning for many students, as most people have different learning styles, personalities, abilities, etc. When users do not have an emotional connection to the content at hand, they become bored, which increases friction to learning and stress for students. Each user may have a predefined profile that may be based on user input preferences or past experience of how the user responded to different interventions. Therefore, each presentation can be customized for each individual. One person may learn better in a more formal school-like virtual environment, while another may learn better in a virtual park-like environment.

[0165] One embodiment may address this problem by creating an intelligent virtual environment. In this embodiment, the system may be used to enhance learning in a classroom or seminar setting. A user profile may contain preferences, such as topics of interest, music, food, background images, culture, personal history, age group, social class, personality, learning type, etc. The system may generate further user profile data that may be used to continually update the user profile based on the user's interactions in the virtual environment and responses to interventions. Thus, the system continually learns from each user and uses this data to further improve itself, better understand each user, and increase its level of empathy.

[0166] User learning style is also useful because it determines how each user learns best to improve their learning efficiency.

[0167] An administrator (e.g., a teacher) can define the content to be delivered to the class. The system can analyze the content and user profile of each student and output multiple presentation media that best matches each student's user profile. As a result, each student may receive a different presentation of similar information. By finding the presentation medium that best suits each student, students can connect with their specific learning style, increase emotional connection with the content, and promote and enhance learning.

[0168] Processing content based on user profiles may include summarizing the content in relatively different ways to suit different students. For some students, the system may use words that are more emotional or resonate better with their specific profiles, while for other students, more technical and specific words may be used. The system may also create or select examples of specific situations that resonate with students in a better way. In addition, the type of visual media selected may vary from person to person. For example, if a student is defined as a visual learner, the system may create a short film and / or select or mix films from different sources to form a more effective learning presentation. For another user who is more analytical and prefers reading, he may be provided with text or video clips with more analytical examples. Individual modifications can also be made in the virtual environment, including changes in color, background, sound, atmosphere, etc., to enhance the learning experience of each student.

[0169] In some embodiments, the system may modify certain aspects of the content to be delivered based on reading user input in substantially real time. For example, if the user appears confused or expresses doubt, the system may read the content and ask the user what, if anything, he can do to improve his experience, and provide options or space for the user to describe his situation. The system may further adjust certain parts of the content, visual, audio, etc., when changes in user behavior are detected in real time. The presentation may be in the form of video, audio, tactile experience, and the virtual environment may be fully modified to present the course content being conveyed, and may take the form of augmented reality and / or virtual reality, etc. The system is not necessarily a virtual environment, but a program that provides these different types of content presentation, one of which is for presentation in a 3D virtual environment, but is not limited to this. The system may also provide this information to teachers or parents to take additional intervention measures to ensure that the child understands what is being learned.

[0170] The system may further categorize students with similar learning styles and provide courses in similar or identical media. This also helps promote group learning and interaction, as users with similar learning styles may benefit from each other's interpretation and processing of courses. Users can be categorized into one or more learning styles (e.g., visual or auditory) and receive similar virtual environments and interventions to meet the needs of users in the same category.

[0171] Instructions executable by one or more processors may be stored on a non-transitory computer-readable medium. Therefore, whenever a computer-implemented method is described in this disclosure, the disclosure should also be understood to describe a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, configure and / or cause one or more processors to perform the computer-implemented method. Examples of non-transitory computer-readable media include RAM, ROM, solid-state storage media (e.g., solid-state drives), optical storage media (e.g., optical disks), and magnetic storage media (e.g., hard drives). The non-transitory computer-readable medium may be part of the memory of a computer system or separate from any computer system.

[0172] It should be understood that in the above description of exemplary embodiments, in order to simplify the present disclosure and aid in understanding one or more of the various inventive aspects, various features are sometimes combined in a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than those explicitly recited in each claim. On the contrary, as reflected in the following claims, the inventive aspects lie in less than all the features of a single preceding disclosed embodiment. The use of the word "or" in the specification is meant to be non-exclusive and should be interpreted to include each individual element alone or in any combination thereof. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim independently serving as a separate embodiment of the present disclosure.

[0173] In addition, although some embodiments described herein include some but not all features included in other embodiments, it should be understood by those skilled in the art that combinations of features of different embodiments are also within the scope of this disclosure and constitute different embodiments. For example, in the following claims, any of the described embodiments can be used in any combination.

[0174] Thus, while certain embodiments have been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present disclosure, and it is intended that all such changes and modifications be included within the scope of the present disclosure. For example, functions may be added or deleted from the block diagrams, or operations may be interchanged between functional blocks. The method steps described may be added or deleted within the scope of the present disclosure.

[0175] The subject matter disclosed above is considered to be illustrative rather than restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other implementations that fall within the true spirit and scope of the present disclosure. The use of the term "or" in this specification means one or more of any available options, and does not have to be a mutually exclusive list. Therefore, to the maximum extent permitted by law, the scope of the present disclosure will be determined by the broadest permissible interpretation of the appended claims and their equivalents, and should not be restricted or limited by the foregoing detailed description. Although various embodiments of the present disclosure have been described, it will be apparent to those of ordinary skill in the art that more embodiments and implementations are possible within the scope of the present disclosure. Therefore, the present disclosure is not limited.

Claims

1. A method comprising: The interactive virtual conference platform stores the plurality of virtual environments in one or more virtual environment systems; The interactive virtual conference platform stores a plurality of context scenarios in the one or more virtual environment systems; The interactive virtual conference platform stores a plurality of emotional cues in the one or more virtual environment systems; receiving, at the interactive virtual conference platform, a plurality of requests to join a virtual conference; Connecting a plurality of sessions corresponding to the plurality of requests through the interactive virtual conference platform, wherein each session includes input data, the input data includes at least one or more video or audio streams, and the at least one or more video or audio streams together form a plurality of video or audio streams; The server of the interactive virtual conference platform analyzes the multiple video streams or audio streams; The interactive virtual conference platform automatically detects contextual scenarios or one or more emotional clues from the input data in the plurality of contextual scenarios; In response to detecting the contextual scenario or emotional clue, the interactive virtual conference platform selects an intervention from an intervention database based on the analyzed input data, the detected contextual scenario, or the detected one or more emotional clues; In response to detecting the contextual scenario, reading, by the interactive virtual conference platform, the intervention from the intervention database; and The virtual conference is intervened by the interactive virtual conference platform based on the intervention read from the intervention database, wherein the intervention comprises at least one change to an output audio signal or an output video signal.

2. The method of claim 1, further comprising: One or more data sets corresponding to the multiple context scenarios are input into one or more neural networks through the interactive virtual conference platform.

3. The method according to claim 2, characterized in that The one or more neural networks include one or more of a convolutional neural network (CNN) and a recurrent neural network (RNN).

4. The method of claim 2, further comprising: Feedback regarding the intervention is received and the feedback is applied to the one or more neural networks to train the one or more neural networks for future interventions.

5. The method according to claim 4, characterized in that The feedback includes a ranking from at least one user.

6. The method according to claim 5, characterized in that The feedback includes physical reactions from one or more users detected via the video or audio stream.

7. The method according to claim 1, characterized in that The intervention includes at least one of a change to a virtual camera angle, a lens size, and a camera motion.

8. The method according to claim 1, characterized in that The intervention includes one or more of changing the brightness of the output video signal, changing the pitch of the output audio signal, and changing the hue of the output video signal.

9. The method according to claim 1, characterized in that Selecting, by the interactive virtual meeting platform, the intervention from the intervention database in response to detecting the contextual scenario further includes selecting the intervention based on one or more user profiles.

10. The method according to claim 9, characterized in that The one or more user profiles include one or more user settings that associate contextual scenarios with preselected intervention criteria.

11. A computer system comprising: Multiple client video conferencing applications; an interactive video conferencing platform configured to receive one or more of the video streams or audio streams from the plurality of client video conferencing applications, wherein the interactive video conferencing platform is configured to analyze the one or more video streams or audio streams and detect a contextual scene based on the analysis; An intervention database of the interactive video conferencing platform, comprising a plurality of interventions, wherein, based on detecting the contextual scenario, the interactive video conferencing platform is further configured to read and implement an intervention corresponding to the contextual scenario; as well as The output signal of the interactive video conferencing platform comprises an output audio signal or an output video signal, wherein the interactive video conferencing platform is configured to modify the output signal based on the intervention corresponding to the context scenario.

12. The computer system of claim 11, wherein the plurality of interventions include changes to one or more camera views and environmental changes.

13. The computer system of claim 12, wherein: The interactive video conferencing platform is also configured to receive feedback regarding the intervention and apply the feedback to one or more neural networks to train the one or more neural networks for application to future interventions.

14. The computer system of claim 11, further comprising: A web interface configured to receive inputs comprising one or more of typing speed, typing volume (cancellation), gestures, amount of speaking time, facial (micro)expressions, mouse / slide speed, geographic location, browser, loading time, FPS / tab focus, meeting title, number of participants, head position, language toxicity, device, or speech cadence, wherein the interactive video conferencing platform is configured to select the intervention based on one or more of the inputs.

15. The computer system of claim 11, wherein: The intervention includes one or more of changing the brightness of the output video signal, changing the pitch of the output audio signal, and changing the hue of the output video signal.

16. The computer system of claim 11, wherein: The intervention corresponding to the context scenario further corresponds to one or more user profiles.

17. The computer system of claim 16, wherein: The one or more user profiles contain user settings for one or more context scenarios associated with preselected intervention criteria.

18. A non-transitory computer-readable medium comprising instructions executable on a processor, the instructions comprising: connecting a plurality of users to a virtual conference based on a plurality of requests to join the virtual conference, wherein each session includes one or more video or audio streams that together form a plurality of video streams or audio streams; analyzing the plurality of video or audio streams to detect a context scene from a plurality of context scenes stored in a scene database; automatically detecting a context scene from among a plurality of context scenes stored in a scene database; selecting an intervention from an intervention database based on the contextual scenario; reading the intervention from the intervention database based on the context scenario; as well as The virtual conference is intervened based on the intervention read from the intervention database, wherein the intervention comprises at least one change to an output audio signal or an output video signal.

19. The non-transitory computer-readable medium of claim 18, further comprising instructions to: One or more neural networks including one or more of a convolutional neural network (CNN) and a recurrent neural network (RNN) are analyzed to detect the context scene.

20. The non-transitory computer-readable medium of claim 18, further comprising instructions for the following operations: the intervention corresponding to the context scenario further corresponds to one or more user profiles.