Personalized digital meeting intelligent agent
By developing a personal digital agent (Ditto) that simulates the user's personality and appearance, the problem of meeting conflicts caused by time zone differences and flexible work arrangements among team members was solved, thus improving the efficiency of virtual meetings.
Patent Information
- Application Number
- CN202580023361.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-09
- Filing Date
- 2025-03-06
- Publication Date
- 2026-11-10
AI Technical Summary
Team members face difficulties attending or effectively participating in virtual meetings due to meeting conflicts caused by global time zone differences and flexible work arrangements, which affects meeting efficiency.
Develop a personal digital agent (Ditto) that is a pre-trained digital proxy for the user, simulating the user's personality, behavior, visual appearance and voice, representing the user in meetings, listening to and answering questions and offering opinions, thereby improving meeting efficiency.
By leveraging visual and auditory similarities, digital agents improve the meeting experience for other attendees and enhance meeting efficiency when team members are absent.
Smart Images

Figure CN123076884A_ABST
Abstract
Description
Background Technology
[0001] A significant characteristic of the work is the highly diverse and globally distributed nature of teams, with team members scattered across different physical locations and time zones. Consequently, business meetings are increasingly being conducted virtually via video and / or audio conferencing. With the widespread use of such virtual communication, scheduling conflicts are more likely to occur, for example, due to more flexible hybrid work arrangements, global time zone differences, and the common challenge of team members being out of the office due to appointments, vacations, etc. Therefore, people often cannot attend all the meetings they are invited to or interested in attending.
[0002] Embodiments have been described in relation to these and other general considerations. Furthermore, although relatively specific problems have been discussed, it should be understood that the embodiments are not limited to solving the specific problems identified in the background section. Summary of the Invention
[0003] To address the aforementioned challenges related to meeting conflicts, aspects of this application relate to a personal digital agent (or "Ditto") pre-trained as a digital proxy for a user (e.g., a human). By adopting the human's persona (e.g., personality, mannerisms, preferences, knowledge, and in some cases, a realistic visual appearance and voice), the personal digital agent can effectively interact on the human's behalf. For example, the digital agent can converse with meeting colleagues and possess a voice, facial expressions, and body language similar to an absent human. Furthermore, the pre-trained digital agent can listen to what the team is saying, ask clarifying questions, answer questions on behalf of the human, and offer perspectives that the human would like the team to consider. Because the digital agent visually resembles, sounds, and acts like the human, it appears remarkably similar to other remote participants, thereby improving the meeting experience for other attendees and increasing meeting efficiency in the absence of human team members.
[0004] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description
[0005] Non-limiting and non-exhaustive examples are described with reference to the following figures.
[0006] Figure 1 An overview of example systems in which personal digital agents can be implemented according to various aspects of this disclosure is shown.
[0007] Figure 2An example virtual meeting interface that can realize a personal digital agent according to various aspects of this disclosure is shown.
[0008] Figures 3A-3C An overview of an example conceptual architecture for implementing a personal digital agent, based on the aspects described herein, is shown.
[0009] Figures 4A-4B An overview of an example conceptual architecture for implementing a state machine for a personal digital agent, based on the aspects described herein, is shown.
[0010] Figures 5A-5C An overview of example method 500 for implementing a personal digital intelligent agent is shown, based on the aspects described herein.
[0011] Figure 6 This is a block diagram illustrating example physical components of a computing device that can be used to practice various aspects of this disclosure.
[0012] Figure 7 It is a simplified block diagram of a computing device that can be used to implement various aspects of the present disclosure.
[0013] Figure 8 This is a simplified block diagram of a distributed computing system in which the various aspects of this disclosure can be implemented. Detailed Implementation
[0014] In the following detailed description, reference is made to the accompanying drawings, which form a part of the description, and specific embodiments or examples are illustrated by way of example in the drawings. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from this disclosure. Embodiments may be practiced as methods, systems, or devices. Accordingly, embodiments may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be considered limiting, and the scope of this disclosure is defined by the appended claims and their equivalents.
[0015] Various aspects of this application relate to a digital agent (or "Ditto") pre-trained as a digital proxy for a user (e.g., a human). In some aspects, the digital agent can be assigned to meetings on behalf of the user. As used herein, a "meeting" can include any interaction between the digital agent and one or more human users and / or one or more other digital agents, whether the interaction is formal or informal, scheduled or ad-hoc, one-on-one or non-one-on-one. Meetings can be conducted via video and / or audio conferencing platforms, via telephone, via text, or any combination thereof. By adopting the human's role (e.g., personality, mannerisms, preferences, knowledge, and in some cases, realistic visual appearance and voice), the digital agent is able to effectively interact on behalf of the human. For example, a pre-trained digital agent can listen to what the team is saying, ask clarifying questions, answer questions on behalf of the human, and offer perspectives that the human would like the team to consider. More specifically, the digital agent can be trained to detect the participation of other attendees (e.g., through gaze detection), respond appropriately to questions or provide input with or without human interaction, determine the mood and flow of conversation based on environmental data and / or non-voice data (e.g., laughter, clapping, smiling, typing, etc.), and possess other skills. Furthermore, the agent dashboard provides the human with real-time visibility into the meeting, enabling the human to intervene and respond on behalf of the digital agent, directly provide input or answer questions posed by the digital agent or other attendees, preview actions the digital agent will take (including approval or disapproval options), and join and replace the agent at any time. Because the digital agent visually resembles, sounds like, and acts like the human, it appears remarkably similar to other remote participants, thus improving the meeting experience for other attendees and increasing meeting efficiency in the absence of team members.
[0016] In the examples, a generative model (often also referred to as the base model) can be used according to the aspects described in this paper, and this generative model can generate any of a variety of output types (therefore, in some examples, it can be a multimodal generative model). For example, the generative model can include generative transformer models and / or large language models (LLM), generative image models, etc. Example models include, but are not limited to, Megatron-Turing Natural Language Generation Model (MT-NLG), Generative Pre-trained Transformer 3 (GPT-3), Generative Pre-trained Transformer 3.5 Turbo (GPT-3.5-turbo), Generative Pre-trained Transformer 4 (GPT-4), BigScience BLOOM, DALL-E, DALL-E 2, Stable Diffusion, Text-Curie-001, or Jukebox. The following is combined with... Figures 5A-5B The generative ML models shown discuss other examples of these aspects. Examples of existing embedding models include, but are not limited to, sentence-BERT (e.g., s-BERT all-MiniLM-L6-v2 or msmarco-distilbert-base-dot-prod-v3), CPU-optimized and quantized patterns (e.g., ggml-all-MilniLM-L6-v2-f16), Word2Vec, S-RoBERTa, GloVe, FastText, CoVe, ELMO, etc.
[0017] Figure 1 An overview of an example system 100 according to various aspects of this disclosure is shown, in which a personal digital agent (or "avatar agent 142") can be assigned to participate in a communication session, such as a meeting, on behalf of a user who may not be able to attend in person. The personal digital agent can be an intelligent application or processor that utilizes artificial intelligence (AI) methods to provide the functionality described herein. For example, such an intelligent digital agent can utilize a model trained using user profile information to simulate personalized human characteristics (e.g., personas), such as visual appearance, voice, mannerisms, personality, expertise, knowledge, and decision-making traits. Such an intelligent digital agent can also be trained based on meeting profile information indicating that the user has attended one or more previous meetings, previous meetings between a particular group of participants, meeting etiquette, and / or professionalism, etc.
[0018] In various aspects, system 100 may include one or more user devices 102, which may be configured to run or otherwise execute an application. The one or more user devices 102 may include, but are not limited to, laptops, tablets, desktop computers, smartphones, Internet of Things (IoT) devices, etc. The application may have conferencing features (“Conference Application 104”) that allow users to virtually attend various communication sessions, such as video conferencing sessions, audio conferencing sessions, streaming presentation sessions, etc. Non-limiting examples of Conferencing Application 104 include Microsoft® Teams® and Microsoft® Skype. ™ Google® Meet®, Cisco® WebEx®, and Zoom®. In some examples, meeting application 104 may include a web version that runs within a web browser or otherwise executes instructions, a native client version residing on user device 102, and / or a server version that is on or implemented by application server 106.
[0019] One or more user devices 102 may be communicatively coupled to application server 106 via network 108. Network 108 may be a wide area network (WAN), such as the Internet, a local area network (LAN), a radio access network (e.g., RAN), or any other suitable type of network. In some examples, network 108 may be a single public or private network, multiple integrated public or private networks, or a distributed public or private network (e.g., a cloud network). Conferencing application 104 may allow user 110 to virtually access various meetings, such as personal or business meetings, conferences, presentations, etc., which may be hosted on or by application server 106.
[0020] System 100 may also include a database 112 for storing information. Database 112 may be communicatively coupled to application server 106 and / or one or more user devices 102 via network 108, such as... Figure 1 As shown, or via any other suitable means now known or developed hereafter, database 112 may be coupled to application server 106 and / or one or more user devices 102. For example, in this example, database 112 may be directly connected to application server 106, communicatively coupled to application server 106, or integrated as part of application server 106. In various respects, database 112 may be a relational or non-relational database, a graph database, an object-oriented database, or any combination of multiple such databases.
[0021] In another aspect, database 112 may store information associated with one or more user profiles 114, meeting profiles 116, and pre-cached ideas 118. Non-limiting examples of information associated with user profiles 114 include a user's image, a recording of the user's voice, statements made by the user (e.g., social media posts, electronic communications, user-created documents, recorded presentations, etc.), the user's business roles and responsibilities, personal and business calendar information, meeting context from the user (e.g., instructions or input regarding a specific meeting topic, meeting goals or objectives, topics of interest, notes prepared for an upcoming meeting, notes or observations from previous meetings, etc.), personal information (e.g., gender, race, education, place of residence, skills, hobbies, activities, interests, food preferences or restrictions, etc.). In another aspect, user profiles 114 may include information about other people associated with user 110. For example, user profile 114 may include information about team members (e.g., name, birthday, role and responsibilities, professional relationships, collaborative projects, etc.), family members (e.g., name, birthday, relationship, etc.), friends (e.g., name, birthday, connection, shared activities or interests, etc.). In some examples, users can use agent dashboard 124 to enter or update information associated with user profile 114.
[0022] Additionally, database 112 may include information associated with meeting profile 116. Meeting profile 116 may include information about previous or upcoming meetings, such as meeting title, meeting description, meeting agenda, meeting transcript, meeting summary, meeting action items, meeting invitees / decline / acceptance, meeting attendees, etc. In some cases, meeting profile 116 may also include up-to-date information about the status of a specific meeting topic (e.g., a specific project, initiative, event, etc.) and associated topic metrics (e.g., timeline, benchmarks, deliverables, deadlines, action items, etc.).
[0023] In a further example, database 112 may include pre-cached ideas 118. Pre-cached ideas 118 may include pre-trained statements associated with knowledge, expertise, or prepared content that a human (e.g., user 110) would like to convey through avatar agent 142 during a meeting. Additionally or alternatively, pre-cached ideas 118 may include structured statements automatically output by avatar agent 142 when delays associated with artificial intelligence (AI) processing or waiting for a human response cause unnatural or awkward pauses in the meeting conversation. For example, while avatar agent 142 awaits an answer, it may say: “Yes, let me think about it…”; “Still thinking…”; or “Let’s continue, and I will answer your question once <human> is able to reply to me.” In various aspects, different pre-cached ideas 118 may be suitable for different scenarios, and scenario triggers may be associated with one or more pre-cached ideas 118 to enable avatar agent 142 to respond appropriately for a specific scenario. For example, some pre-cached ideas 118 may be associated with a first scene trigger (e.g., "waiting for an answer from a human"), while other pre-cached ideas 118 may be associated with a second scene trigger (e.g., "waiting for output from the model"). In some cases with longer delays, the avatar agent 142 may provide more than one pre-cached idea 118 in a particular scene, for example, by first saying "Let me think for a minute," and then saying "Let's continue, I'll raise my hand when I have an answer." In various ways, the avatar agent 142 may verbally provide pre-cached ideas 118 or provide them in thought bubbles associated with the agent avatar.
[0024] like Figure 1As shown, application server 106 may include or communicate with one or more components for implementing avatar agent 142. In some examples, application server may include conferencing application 104, which may include or communicate with one or more components for implementing avatar agent 142. In each aspect, each component may include software and / or hardware for implementing aspects of the functionality described herein. For example, conferencing monitor 120 may include or communicate with software and / or hardware (e.g., a processor) configured to receive conferencing input. Conferencing input may refer to any raw or processed data collected during a conferencing session, including sound, speech, images, video, text, etc. Conferencing input may be received from devices (e.g., sensors, microphones, cameras, user device 102, etc.), human interactions (e.g., voice or text interactions via agent dashboard 122), software applications (e.g., implementing optical character recognition, gaze detection, speech-to-text, facial recognition, object recognition, etc.). In various aspects, the meeting monitor 120 receives meeting input continuously or at regular time intervals (e.g., every 100 milliseconds (ms), at 10 frames per second (fps), or based on a dynamic window). In a further aspect, the meeting monitor 120 may include or communicate with an agent dashboard 122. (As combined...) Figure 2 More fully, the agent dashboard 122 can provide interface functionality that enables a person associated with the clone agent 142 to directly supervise and control the agent's actions, including fields for text input, a microphone for voice input, and / or controls for joining a meeting or intervening on behalf of the clone agent 142.
[0025] In addition to the meeting monitor 120, the application server 106 and / or the meeting application 104 may also include or communicate with various detection components, including but not limited to the participation detector 124, the environmental data detector 126, and the meeting atmosphere detector 128. In various aspects, the detection components may include or communicate with one or more machine learning models trained to detect various characteristics of the meeting based on meeting input received by the meeting monitor 120. In some aspects, the one or more machine learning models may include one or more base models. For example, the participation detector 124 may be configured or trained to detect when the clone agent 142 participates during the meeting. The participation detector 124 may detect participation based on direct communication with the clone agent 142 (e.g., “Sam’s clone, what do you think?”) or indirect communication (e.g., eye contact towards the digital agent). In a further aspect, the participation detector 124 may detect when the clone agent 142 is asked a direct question (e.g., “Sam’s clone, when is Sam coming back from vacation?”). The environmental data detector 126 can be trained to detect environmental data based on meeting input received by the meeting monitor 120. For example, the environmental data detector can detect nonverbal environmental audio (e.g., laughter, clapping, typing, dog barking, telephone ringing or beeping, etc.), verbal environmental audio (e.g., background dialogue), or environmental visual information (e.g., people or objects in the video background, physical objects, etc.). Based on the environmental data detected by the environmental data detector 126 and / or the meeting input received by the meeting monitor 120, the meeting atmosphere detector 128 can be trained to infer characteristics of the meeting or meeting attendees, such as conversational mood (e.g., rude, argumentative, cheerful, formal, casual, etc.), conversational flow (e.g., brainstorming, reporting, planning, presentation, etc.), attendee identification (e.g., based on voice or facial recognition), attendee engagement (e.g., focused, distracted, unengaged, etc.), nonverbal communication or cues (e.g., interpersonal distance, eye contact, facial expressions, body language, etc.).
[0026] The persona engine 130 can be configured to generate an avatar agent 142 that resembles user 110 (e.g., a human). For example, based on user profile 114, the persona engine 130 can utilize various generative models to simulate user 110's mannerisms, personality, visual characteristics, voice, vocabulary, etc. In various aspects, the visual similarity of the avatar agent 142 to this human can range from highly realistic (e.g., manipulating video recordings of the human using generative models) to fantastical (e.g., creating an avatar of the human using diffusion techniques). In a further aspect, the persona engine 130 can "manipulate" the avatar agent 142 to mimic various gestures and facial features when communicating during meetings. In this way, the avatar agent 142 not only "human-like" but also embodies an actual human who cannot attend the meeting, thereby facilitating more natural interaction with the avatar agent 142 and improving meeting efficiency in the absence of team members.
[0027] Application server 106 and / or conferencing application 104 may also include or communicate with state engine 132, which processes conference input and determines the actions of avatar agent 142. In various aspects, state engine 132 communicates with artificial intelligence (AI) model manager 140 to select and query various ML models to process conference input and output appropriate actions for avatar agent 142. State engine 132 may be associated with a “thinking loop” invoked at regular intervals (e.g., every 100 milliseconds (ms), 10 frames per second (fps), or based on a dynamic window). State engine 132 may be a “stateless machine” in which each frame obtains its state from the previous iteration and outputs its state to the next iteration. Furthermore, each frame obtains conference input and issues a new output, which may include, for example, sending a request to an ML model or a human, and obtaining a response to a previous request. State engine 132 may also be replayable, enabling feedback and fine-tuning of the system. For example, the input, output, state input, and state output of each frame may be recorded and replayed to perform real-time debugging. Such as combination Figure 4B Further described, the state engine 132 may include an initial state machine 134, a listening state machine 136, and a response state machine 138. The initial state machine 134 may be pre-trained using pre-cached ideas 118 to provide context or to pre-set the listening state machine 136. In each aspect, the listening state machine 136 receives a "state input" from the previous frame, receives a meeting input, sends a request for output, receives the output, and sends a "state output" to the next frame. The response state machine 138 may receive a direct question state, send a request for an answer, receive the answer, output the answer, and return to the listening state machine 136.
[0028] Listening state machine 136 and / or responding state machine 138 can communicate with AI model manager 140, which can select and query one or more ML models in response to a request for output. In some examples, AI model manager 140 can communicate with a library of ML models trained for a specific task (e.g., via various application programming interfaces (APIs)) and can select an appropriate ML model based on a request. In other examples, AI model manager 140 can communicate with one or more base (or generative) models that are more generally adapted to provide output for a wide range of tasks. AI model manager 140 can also be adapted to generate cue words for querying models, which may include context in addition to conference input. In various aspects, the context may include information from user profile 114, conference profile 116, conference transcripts or summaries, or any other context used to conditionalize (or pre-set) the ML model to provide appropriate output in response to a request.
[0029] As described above, the AI manager 140, the listening state machine 136, and / or the responding state machine 138 can process input based on a “thinking loop.” For example, a large cue word including role information about user 110 can be generated, and as the meeting continues, the cue word can be updated with additional input and generative judgments about the meeting’s progress. For example, in addition to the meeting input received at each iteration, the cue word can also be updated with a real-time meeting transcript or meeting summary including topics discussed so far. With each iteration of the thinking loop, the updated cue word can be input into one or more base models or other ML models to generate output. In some cases, the model output can indicate no action, and the loop can return to the listening state machine for the next iteration. In other examples, the model can output an action to be performed by the digital agent. This action can then be performed (or “manipulated”) by the digital agent (e.g., raising a hand, displaying a thought bubble, asking a user a question, providing a response, etc.). After this manipulated action, the digital agent can store its proprioception (e.g., a sense of location or movement) as state information for the next iteration.
[0030] In some cases, the model can also output the time pending a response. If the model output does not return within the allocated time, a pre-cached idea (e.g., "still thinking") can be provided to minimize the disruption to the dialogue flow caused by the delay. In one specific example, the model can detect that the digital agent is being asked a direct question. In this case, the thinking loop can proceed to the response state machine 138 to gather or retrieve information (the answer). In other examples, retrieval-enhanced generation (RAG) can be used to obtain the answer from an external source. In other examples, the answer can be returned based on the user profile or other personalized information or stored information. In yet another example, the digital agent may deem it necessary to request direct input from the user.
[0031] like Figure 1 As shown, the system can generate multiple clone agents 142 (e.g., clone agents 142-1 to 142-y), each clone agent 142 corresponding to a specific human (e.g., user 110-1 to user 110-x) based at least in part on user profile 114. As mentioned above, the clone agent 142 generated for a specific human can resemble that human in several ways, such as mannerisms, visual appearance, voice, vocabulary, etc. Additionally, the clone agent 142 may be able to “sense the atmosphere of the meeting” and adjust its actions accordingly. For example, the clone agent 142 can be configured to detect the interactive characteristics of the meeting and dynamically adjust its interaction mode. That is, the clone agent 142 may be able to detect whether the meeting is less interactive (e.g., presentation) or more interactive (e.g., planning, brainstorming) and can dynamically adjust its level of interaction during the meeting. As another example, the clone agent 142 may be able to detect whether the meeting is more formal or more casual. That is, the avatar agent 142 can exhibit more professional behavior during company-wide meetings or meetings with user 110's superiors, while exhibiting more casual behavior during brainstorming meetings with project teams. It should be understood that the avatar agent 142 can be continuously trained and fine-tuned to improve its human-like performance during meetings, thereby creating a richer and more efficient meeting experience for human attendees.
[0032] Figure 2 Describes the use of conferencing applications (e.g., Figure 1The conferencing application 104 hosts a conferencing interface 200, which runs on or is accessed by a computing device (e.g., user device 102) associated with or by the first absent user 206A (“Frank”) of the video conference 202. As shown, the user interface 200 displays a frame of the conference 202. While in this example, the conference 202 is visible on the computing device of the first absent user 206A, the first absent user 206A may not be directly participating in the conference 202. In other examples, the conference 202 may not even be open on the computing device of the first absent user 206A.
[0033] like Figure 2 As further shown, the present user 204 (“Asia”) is displayed in the first pane 210 of the user interface 200, the first clone agent 206B (corresponding to the first absent user 206A “Frank”) is displayed in the second pane 216 of the user interface 200, and the second clone agent 208 (corresponding to the second absent user “Sam”) is displayed in the third pane 214 of the user interface 200. In this example, the first clone agent 206B has a realistic visual similarity to the first absent user 206A (“Frank”), while the second clone agent 208 has a more fantastical (or cartoonish) similarity to the second absent user. Figure 2 As further shown, the second agent 208, along with a thought bubble 228, is displayed, informing other attendees that the second agent 208 has determined that the meeting involves planning an event and that the current conversation topic relates to selecting a date for that event. On the other hand, when the digital agent is unable to answer a question raised during the meeting, the thought bubble 228 may display, for example, "I do not currently have an answer to this question, but I will follow up and provide an answer after the meeting." Additionally, the second agent 208 is raising its hand 230, indicating to other attendees that the second agent 208 wishes to participate in the meeting conversation. Although not shown, the second agent 208 may have determined to convey dates that are unavailable to a second absent user, suggest dates that are available to a second absent user, or otherwise comment on the current topic.
[0034] As shown in the figure, a clone dashboard 212 is provided through the meeting interface 200. In other examples, the clone dashboard 212 may operate in a separate interface that can be accessed on demand and / or notifies the first absent user 206A when relevant meeting information or issues are detected during meeting 202 (e.g., by displaying a pop-up window). The clone dashboard 212 provides non-limiting examples of interface functionality that enable the first absent user 206A to have varying degrees of control over the first clone agent 206B during meeting 202. For example, a clone action scrollbar 222 may provide real-time updates on what the first clone agent 206B is thinking (e.g., making a request to the model) and / or what it is about to perform (e.g., an action proposed based on the model's output).
[0035] In some respects, the veto button 232 enables the first absent user 206A to intervene and prevent the first clone agent 206B from taking the proposed action. In other respects, the text field 224 enables the first absent user 206A to directly input a response (or action) for the first clone agent 206B. Other non-limiting functions may include a button 218 for retrieving real-time transcriptions of the meeting 202, or a join button 220 that enables the first absent user 206A to join the meeting 202 in place of the first clone agent 206B. It should be understood that the described examples of interface functions are non-limiting, and other functions may be provided, such as buttons for viewing meeting summaries, enabling the microphone to speak into the meeting, etc.
[0036] Figures 3A-3C An overview of an example conceptual architecture 300 for implementing a personal digital intelligent agent is shown, based on the aspects described herein.
[0037] like Figure 3AAs shown in the conceptual architecture 300, input 302 (including user data 312 and meeting input 314) can be fed into the clone component 304, which includes clone logic 316. Based on the processing of input 302, clone rendering component 306 generates a digital agent with a role associated with a specific user. For example, the digital agent can be rendered to resemble the visual appearance, behavior, personality, voice, knowledge, expertise, vocabulary, etc., of that specific user. In this way, the digital agent is not only designed to act on behalf of that specific user during the meeting, but also to present the image of that specific user in the meeting. As the meeting progresses, clone rendering component 306 and clone component 304 form a processing loop, so that the rendering of the digital agent is synchronized with the reception and processing of incoming input 302, which includes detected conditions, judgments, proposed actions, responses, answers, etc., output by the model associated with clone logic 316. In various aspects, the rendering of actions can be based on the user's role. For example, audio rendering of actions can include a digital agent simulating the human's voice to ask or answer questions; visual rendering of actions can include a digital agent using simulated human gestures or facial expressions to ask or answer questions; and audiovisual rendering can include a combination of the human's audiovisual features. User loop 310 can be associated with a sandbox and / or dashboard for training or controlling the digital agent before, during, and / or after a meeting. For example, user loop 310 enables pre-training (e.g., pre-cached ideas, preparation for a specific topic or meeting), testing (e.g., evaluating performance based on pre-recorded meetings), feedback (e.g., fine-tuning agent interactions, evaluating the relevance and / or accuracy of responses, fine-tuning character rendering, etc.) and / or control or redirection during the meeting. During the meeting, absent users can decide to join at any time. In some aspects, the clone presentation component 308 enables human users to join the meeting in a "picture-in-picture" (PIP) format within the same pane as the digital agent. In other respects, the clone presentation component 308 enables human users to completely replace the digital agent and enter the meeting as a participant (or attendee) within the meeting interface (e.g., within the pane occupied by the digital agent).
[0038] Figure 3BThe types of user data 318 and user context 320 that can be received as user input 312 are illustrated. For example, user data type 318 may include, but is not limited to, notes provided by the user to prepare for a meeting, user profile information (e.g., user profile 114) that may be provided through a graph database or other database (e.g., database 112), and / or long-term memory information (e.g., information collected and stored by the digital agent). For example, long-term memory may include a history of conversation topics covered during the meeting and decisions made (e.g., discussing Mexican food and deciding not to choose it), proprioception (e.g., the digital agent's sense of its location and movement), previously collected information (e.g., obtained through retrieval enhancement generation), pre-cached ideas, etc. User context 320 may include, but is not limited to, topic-based notes (e.g., user opinions or inputs on specific topics that may arise in the meeting), clone-injected notes (e.g., information that the digital agent considers relevant during the meeting), and direct user input (e.g., user responses to clone notifications that involve requests, queries, topics that have appeared, or information that is considered relevant).
[0039] Figure 3C A non-limiting example of meeting input 314 is shown. As described above, meeting input can be received continuously or at regular time intervals. Types of meeting input 314 that can be received and / or detected include audio input 322, video input 324, text input 326, and / or generative output 328. For example, audio input 322 may include, but is not limited to, real-time audio conference recordings, ambient audio or non-speech audio, pre-recorded (stored) audio, etc. Video input 324 may include, but is not limited to, real-time video conference recordings, pre-recorded (stored) video, etc. Text input 326 may include, but is not limited to, speech-to-text conference transcripts, generated meeting summaries, previous meeting transcripts or summaries, documents, social media posts, text, chat, etc. Generative output 328 may include, but is not limited to, judgments made by one or more ML models, such as participation detection (e.g., determining that other attendees in the meeting are interacting with the digital agent), gaze detection (e.g., detecting gazes toward the digital agent), and atmosphere detection (e.g., judgments about conversational mood, conversational flow, level of interaction, etc.). It should be understood that the system can receive any type of input, such as raw data, processed data, generative data, stored data, retrieval-enhanced data, etc.
[0040] Figures 4A-4B An overview of an example conceptual architecture 400 for implementing a state machine 402 of a personal digital agent, based on the aspects described herein, is shown.
[0041] like Figure 4A As shown, the clone logic 316 includes combining Figure 4B The state machine 402 is further described (e.g., similar to state engine 132). In addition to receiving conference input 314 (see...) Figure 3A In addition to the aforementioned ideas, the clone logic 316 can receive training data, such as pre-cached ideas 404. Similar to pre-cached ideas 118, pre-cached ideas 404 can include pre-trained statements associated with the knowledge, expertise, or prepared content that the user wishes to convey through the digital agent during the meeting. Additionally or alternatively, pre-cached ideas 404 can include structured statements for automatic output by the digital agent when delays associated with artificial intelligence (AI) processing or waiting for human responses cause unnatural or awkward pauses in the meeting conversation. In various aspects, different pre-cached ideas 404 may be suitable for different scenarios, and scenario triggers can be associated with one or more pre-cached ideas 404 to enable the digital agent to respond appropriately to a specific scenario. In some cases with longer delays, the digital agent may provide more than one pre-cached idea 404 in a given scenario.
[0042] The clone logic 316 can also incorporate generative inputs from various detection components, including an engagement detector 406, an environmental data detector 408, and a meeting atmosphere detector 410. In various aspects, for example, the detection components may include or communicate with one or more machine learning models trained to detect various characteristics of the meeting based on the meeting input 314. In various aspects, the one or more machine learning models may include one or more base models. For example, engagement detector 406 (e.g., the same as or similar to engagement detector 124) may be configured or trained to detect when a digital agent is engaged during the meeting. Engagement detector 406 may detect engagement based on direct communication with the digital agent (e.g., “Sam’s clone, what do you think?”) or indirect communication (e.g., eye contact towards the digital agent). In a further aspect, engagement detector 406 may detect when a digital agent is asked a direct question (e.g., “Sam’s clone, when is Sam coming back from vacation?”). Environmental data detector 408 (e.g., the same as or similar to environmental data detector 126) may be trained to detect environmental data based on the meeting input 314. For example, the environmental data detector can detect nonverbal environmental audio (e.g., laughter, clapping, typing, dog barking, telephone ringing or notification sounds, etc.), verbal environmental audio (e.g., background dialogue), or environmental visual information (e.g., people or objects in the video background, physical objects, etc.). Based on the environmental data detected by the environmental data detector 408, the meeting atmosphere detector 410 (e.g., the same as or similar to the meeting atmosphere detector 128) can be trained to infer the characteristics of the meeting or meeting attendees, such as conversational mood (e.g., rude, argumentative, cheerful, formal, casual, etc.), conversational flow (e.g., brainstorming, reporting, planning, presentation, etc.), attendee identification (e.g., based on voice or facial recognition), attendee engagement (e.g., focused, distracted, unengaged, etc.), nonverbal communication or cues (e.g., interpersonal distance, eye contact, facial expressions, body language, etc.).
[0043] As described above regarding user loop 310, clone logic 316 and / or state machine 402 can receive human input at any time (e.g., human approval 412). In all respects, human intervention and / or supervision of the digital agent can range from passive supervision to full control, and can be received before, during, and / or after the meeting.
[0044] The clone logic 316 and / or state machine 402 may include or communicate with an AI model manager 414 (e.g., the same as or similar to AI model manager 140), which may select and query an appropriate ML model in response to a request for output. In some examples, the AI model manager 414 may communicate with a library of ML models trained for a specific task (e.g., via various APIs) and may select an appropriate ML model based on a request. In other examples, the AI model manager 414 may communicate with one or more base (or generative) models that may be more generally adapted to provide output for a wide range of tasks. The AI model manager 414 may also be adapted to generate cue words for querying the model, which may include context in addition to meeting input. In various aspects, the context may include personal information, meeting information, or any other context used to conditionalize (or pre-set) the ML model to provide an appropriate output in response to a request. When the ML model outputs an action to be taken by the digital agent, the manipulation component 416 may cause the digital agent to perform that action (e.g., raise a hand, display a thought bubble, verbally give an answer, etc.). Subsequently, the proprioceptive component 418 can store the digital agent's senses of its position or movement and provide them as state information for the next iteration.
[0045] Figure 4B A conceptual architecture 400 of state machine 402 is shown. State machine 402 (e.g., the same as or similar to state engine 132) can be coupled with AI model manager 414 ( Figure 4AThe system communicates to select and query various ML models to process conference inputs and output appropriate actions for the digital agent. State machine 402 can implement a "thinking loop" called at regular intervals (e.g., every 100 milliseconds (ms), 10 frames per second (fps), or based on a dynamic window). State machine 402 can be a "stateless machine," where each frame obtains its state from the previous iteration and outputs its state to the next iteration. Furthermore, each frame obtains conference inputs and issues new outputs, which may include, for example, sending a request to an ML model or a human and obtaining a response to a previous request. State machine 402 can also be replayable, enabling fine-tuning of the system. For example, the inputs, outputs, state inputs, and state outputs of each frame can be recorded and replayed to perform real-time debugging. As shown, state machine 402 includes an initial state machine 420, a listening state machine 422, and a responding state machine 424. In various aspects, the initial state machine 420 can be preset with pre-buffered ideas 426 to provide context to the listening state machine 422. For example, the listening state machine 422 can obtain the initial context from the initial state machine 420, obtain the "state input" 428 from the previous frame, listen to the meeting input 430, obtain the meeting input 432, send a request for output 434 based on the meeting input, obtain the output 436, process the output 438, and send the "state output" to the next frame 440. As a special case, when a direct question is detected (e.g., obtaining the direct question state 442), thinking can cycle into the response state machine 424. Then, the response state machine 424 can send a request for an answer 444 (e.g., a RAG request), periodically display the thinking progress 446 (e.g., when a delay might cause unnatural pauses in the conversation), obtain an answer 448, output the answer 450, and return to the listening state 452 associated with the listening state machine 422.
[0046] Further referencing the think loop, a large cue word can be generated that includes role information about absent users, and this cue word can be updated with additional inputs and generative judgments about the meeting's progress as the meeting continues. For example, in addition to the meeting inputs received at each iteration, the cue word can be updated with a real-time meeting transcript or summary that includes topics discussed so far. With each iteration of the think loop, the updated cue word can be fed into one or more base models or other ML models to generate outputs. In some cases, the model output can indicate no action, and the loop can return to the listening state machine for the next iteration. In other examples, the model can output an action to be performed by the digital agent. This action can then be performed (or "manipulated") by the digital agent (e.g., raising a hand, displaying a thought bubble, asking a user a question, providing a response, etc.). In various ways, the action can be manipulated to mimic a user's behavior, gestures, or characteristics. After this manipulated action, the digital agent can store its proprioception (e.g., a sense of location or movement) as state information for the next iteration.
[0047] Figures 5A-5C An overview of an example method 500 for implementing a personal digital intelligent agent, based on the aspects described herein, is shown.
[0048] Figure 5A Method 500A is illustrated, which begins with an instantiation operation 502, in which a digital agent is instantiated to represent user actions using a user's (e.g., a person's) persona. For example, one or more generative models can instantiate the digital agent based on personal attributes and information associated with the user. Using a model trained with user profile information, the digital agent can be instantiated to simulate personalized human characteristics (e.g., personas), such as visual appearance, voice, mannerisms, personality, expertise, knowledge, and decision-making traits. The digital agent can also be trained based on meeting profile information representing one or more previous meetings the user has attended, previous meetings among a specific group of attendees, meeting etiquette, and / or professionalism, etc.
[0049] At instruction receiving operation 504, an instruction to assign a digital agent to the meeting on behalf of the user can be received. In some aspects, the digital agent can be automatically assigned to the meeting in response to determining that the user is unable to attend. In other aspects, the user can choose to have a digital agent assigned to the meeting on their behalf. Because the digital agent adopts the human's role (e.g., personality, behavior, preferences, knowledge, and in some cases, a realistic visual appearance and voice), the digital agent is able to effectively interact on behalf of the human in the meeting.
[0050] At monitoring operation 506, the digital agent can monitor the meeting. For example, the digital agent can invoke a listening state machine to monitor the meeting input continuously or at regular time intervals. In a loop, the listening state machine obtains the "state input" from the previous frame, receives the meeting input, sends a request for output, receives the output, and sends the "state output" to the next frame.
[0051] At operation 508, it is determined whether conference input has been received. Non-limiting examples of conference input types that can be received and / or detected include audio input, video input, text input, and / or generative output. Audio input may include, but is not limited to, real-time audio conference recordings, ambient or non-speech audio, pre-recorded (stored) audio, etc. Video input may include, but is not limited to, real-time video conference recordings, pre-recorded (stored) video, etc. Text input may include, but is not limited to, speech-to-text conference transcripts, generated conference summaries, previous conference transcripts or summaries, documents, social media posts, text, chat, etc. If no conference input has been received, the method may return to monitoring operation 506. If conference input has been received, the method may proceed to processing operation 510.
[0052] At processing operation 510, the meeting input can be processed by one or more machine learning models. In some examples, a task-specific ML model (e.g., an ML model trained to detect gaze) can be selected based on the meeting input. In other examples, a base model trained on a wide range of topics for a broad range of tasks can be selected based on the meeting input (e.g., a base model for determining conversational sentiment). In further examples, cue words including role information about user 110 can be generated, and as the meeting continues, the cue words can be updated with additional input and generative judgments about the meeting's progress. For example, in addition to the meeting input received at each iteration, the cue words can be updated with a real-time meeting transcript or summary including topics discussed so far. With each iteration of the think loop, the updated cue words can be fed into one or more base models or other ML models to generate output.
[0053] At the output receiving operation 512, model output can be received from one or more ML models. Model output can include any of a variety of output types, such as answers to questions, detection of conditions (e.g., ambient audio, digital agent participation), judgments about conditions (e.g., conversational emotions), actions (e.g., raising a hand, displaying a thought bubble), questions (e.g., clarifying a question), etc.
[0054] At operation 514, it can be determined whether the model output indicates that the digital agent should take an action. In some cases, the model output may indicate that no action should be taken, and the loop can return to the listening state machine at monitoring operation 506. Otherwise, the method can proceed to operation 516.
[0055] At operation 516, the model output can be used to determine how the digital agent should respond (or what action it should perform). For example, if the model output indicates that someone has asked the digital agent a question in a rude manner, it can be determined that the digital agent should smile and reply "Please ask politely" instead of requesting an answer to the question. As another example, if the model output is a response to a query, it can be determined that the digital agent should raise its hand. In various aspects, determining the action includes determining the rendering used to manipulate the digital agent to perform that action (e.g., raise its hand). It should be understood that the model output can suggest any of a large number of possible responses from the digital agent. In fact, the model output itself can be fed into another model to determine the appropriate response action for that output.
[0056] At actuation operation 518, the system can manipulate the digital agent to perform a predetermined action. Continuing the example above, the generative model can cause the digital agent to verbally utter the phrase "Please ask questions politely," or it can cause the digital agent to raise an arm and point its palm towards the audience. The method can continue to loop back to monitoring operation 506 until it receives an indication that the meeting has ended or that the digital agent has been authorized to leave the meeting (e.g., replaced by the user).
[0057] Figure 5B Method 500B is shown, which illustrates the following regarding Figure 5A Further details regarding the instantiation operation 502.
[0058] At receiving operation 502A, user data and / or context may be received. Non-limiting examples of user data may include personal information (e.g., gender, race, educational background, place of residence, skills, hobbies, activities, interests, food preferences or restrictions, etc.), images of the user, voice recordings of the user, statements made by the user (e.g., social media posts, electronic communications, user-created documents, recorded presentations, etc.), the user's business roles and responsibilities, personal and business calendar information, etc. For example, non-limiting examples of context may include instructions or input regarding a specific meeting topic, meeting objectives or purposes, topics of interest, preparation notes for an upcoming meeting, notes or observations from previous meetings.
[0059] At generation operation 502B, intelligent agent roles can be generated. For example, roles that reflect the user's behavior, voice, personality, vocabulary, and gestures can be generated based on user data and context.
[0060] At generation operation 502C, a visual rendering of the digital intelligent agent can be generated. In some respects, the visual rendering can range from highly realistic (e.g., using generative models to manipulate video image recordings of users) to fantastical (e.g., using diffusion techniques to render an avatar similar to the user).
[0061] At Operation 502D, an agent sandbox can be provided. The agent sandbox can be an interactive interface, for example, for accepting user input and feedback on digital agent roles and / or rendering.
[0062] At test operation 502E, digital agents can be tested. For example, meeting recordings can be used to test model processing and agent responses.
[0063] At receive operation 502F, feedback can be received based on the test sandbox.
[0064] At update operation 502G, agent roles, rendering, and / or model processing can be updated and fine-tuned via the agent sandbox.
[0065] In all aspects, once the digital agent is prepared using the agent sandbox, this method can return to Figure 5A Instantiation operation 502.
[0066] Figure 5C Method 500C is shown, which illustrates the following regarding Figure 5A Further details of operations 510-514.
[0067] At generation operation 509, cue words based on meeting input and context can be generated. For example, a large cue word including information about the user's role can be generated, and as the meeting continues, the cue word can be updated with additional input and generative judgments about the meeting's progress. For example, in addition to the meeting input received at each iteration, the cue word can be updated with a real-time meeting transcript or meeting summary including topics discussed so far. With each iteration of the think loop, the updated cue word can be input into one or more base models or other ML models to generate output.
[0068] At processing operation 511, the cue words can be processed by one or more ML models to generate model output. Based on the cue words, an ML model trained for a specific task can be selected (e.g., an ML model trained to detect gaze). In other examples, a base model trained on a wide variety of topics for a wide range of tasks can be selected based on the meeting input (e.g., a base model used to determine the sentiment of the conversation).
[0069] At the output receiving operation 512, model output can be received from one or more ML models. Model output can include any of a variety of output types, such as answers to questions, condition detection (e.g., ambient audio, digital agent interaction), judgments about conditions (e.g., conversational emotions), actions (e.g., raising a hand, displaying a thought bubble), questions (e.g., clarifying a question), etc.
[0070] At operation 513A, it can be determined whether the output is relevant. In some examples, the model output can be fed into another ML model to determine relevance. In various ways, relevance can be based on the difference between the model output and conference attributes (e.g., the topics discussed, topic content, conference transcripts, conference abstracts, etc.). If the model output is not reasonably relevant to the conference, it can be determined that the model output is irrelevant, and the method can return to monitoring operation 506. If the model output is not irrelevant, the method can proceed to determination operation 513B.
[0071] At step 513B, it can be determined whether the model output indicates that the digital agent is explicitly participating in the meeting. For example, if the meeting input is a statement directly naming the digital agent, the model output can indicate that the digital agent is explicitly participating. If the digital agent is explicitly participating, the method can proceed to step 516 to determine how the digital agent should respond. If the digital agent is not explicitly participating, the method can proceed to step 513C.
[0072] At operation 513C, it can be determined whether environmental data has been detected. Non-limiting examples of environmental data include nonverbal environmental audio (e.g., laughter, clapping, typing, dog barking, telephone ringing or alert tones, etc.), verbal environmental audio (e.g., background dialogue), or environmental visual information (e.g., people or objects in a video background, physical objects, etc.). If environmental data is detected, the method can proceed to operation 513D. If environmental data is not detected, the method can proceed to operation 514, where it can be determined whether the digital agent should take an action.
[0073] At operation 513D, the characteristics of the meeting or its attendees can be determined based on the detected environmental data. Meeting characteristics may include conversational mood (e.g., rude, argumentative, cheerful, formal, casual, etc.), conversational flow (e.g., brainstorming, reporting, planning, presentation, etc.), attendee identification (e.g., based on voice or facial recognition), attendee engagement (e.g., focused, distracted, unengaged, etc.), and nonverbal communication or cues (e.g., interpersonal distance, eye contact, facial expressions, body language, etc.).
[0074] At operation 513E, it can be determined whether the digital agent is implicitly involved based on the determined meeting characteristics. For example, if the model output indicates the presence of a gaze toward the digital agent, it can be inferred that the digital agent is implicitly involved. If it is determined that the digital agent is implicitly involved, the method can proceed to operation 516, where it is determined how the digital agent should respond. If it is determined that the digital agent is not implicitly involved, the method can proceed to operation 514, where it is determined whether the digital agent should take an action.
[0075] It should be understood that operations 502-518 are described for the purpose of illustrating the method and system and are not intended to limit the disclosure to a particular sequence of steps. For example, the operations may be performed in a different order and more or fewer operations may be performed without departing from the disclosure.
[0076] Figure 6-8 The related descriptions discuss various operating environments in which various aspects of this disclosure can be practiced. However, in conjunction with... Figure 6-8 The devices and systems shown and discussed are for illustrative purposes only and are not intended to limit the numerous computing device configurations that can be used to practice the various aspects of this disclosure described herein.
[0077] Figure 6 This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 600 that can be used to practice the aspects of this disclosure. The computing device components described below are applicable to the aforementioned computing device, including those combined with the above. Figure 1 One or more computing devices are discussed (e.g., user device 102, application server 106). In a basic configuration, computing device 600 may include at least one processing unit 602 and system memory 604. Depending on the configuration and type of the computing device, system memory 604 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memory.
[0078] System memory 604 may include operating system 605 and one or more program modules 606 suitable for running software application 620, such as one or more components supported by the system described herein. As an example, application 620 (e.g., a conferencing application) may run various modules to perform the functions described herein, such as a meeting monitor 624, an participation detector 626, a role manager 628, and / or a state machine 630. For example, operating system 605 may be adapted to control the operation of computing device 600.
[0079] Furthermore, embodiments of this disclosure can be practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. This basic configuration is... Figure 6 The components within the dashed line 608 are shown. The computing device 600 may have additional features or functions. For example, the computing device 600 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 6 The image is shown by removable storage device 609 and non-removable storage device 610.
[0080] As described above, multiple program modules and data files can be stored in system memory 604. When executed on processing unit 602, program module 606 (e.g., application 620) can perform processes including, but not limited to, the aspects described herein. Other program modules that can be used according to aspects of this disclosure may include a metric monitor, a definition database, etc.
[0081] Furthermore, embodiments of this disclosure can be practiced in circuits including discrete electronic components, packages containing logic gates or integrated electronic chips, circuits utilizing microprocessors, or on a single chip including electronic components or a microprocessor. For example, embodiments of this disclosure can be practiced via a system-on-a-chip (SoC), wherein... Figure 6 Each or many of the components shown can be integrated onto a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “programmed”) onto a chip substrate as a single integrated circuit. When operating via the SoC, the capabilities described herein regarding the client switching protocol can be operated via application-specific logic integrated with other components of the computing device 600 on the single integrated circuit (chip). Embodiments of this disclosure can also be practiced using other techniques capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, embodiments of this disclosure can be practiced within a general-purpose computer or in any other circuit or system.
[0082] The computing device 600 may also have one or more input devices 612, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. It may also include one or more output devices 614, such as a display, speaker, printer, etc. The above devices are examples, and other devices may be used. The computing device 600 may include one or more communication connections 616 that allow communication with other computing devices 650. Examples of suitable communication connections 616 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel ports, and / or serial ports.
[0083] As used herein, the term computer-readable medium can include computer storage media. Computer storage media can include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 604, removable storage device 609, and non-removable storage device 610 are examples of computer storage media (e.g., memory storage). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computing device 600. Any such computer storage medium can be part of computing device 600. Computer storage media does not include carrier waves or other propagated or modulated data signals.
[0084] Communication media can be embodied in computer-readable instructions, data structures, program modules, or other data (such as carrier waves or other transmission mechanisms) in modulated data signals, and include any information transmission medium. The term "modulated data signal" can describe a signal having one or more characteristics set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.
[0085] Figure 7 A system 700 is illustrated, which may be, for example, a mobile computing device such as a mobile phone, smartphone, wearable computer (such as a smartwatch), tablet computer, laptop computer, etc., and embodiments of the present disclosure can be practiced with the system 700. In one embodiment, the system 700 is implemented as a "smartphone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 700 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and cordless phone.
[0086] In its basic configuration, this type of mobile computing device is a handheld computer with both input and output elements. System 700 typically includes a display 705 and one or more input buttons that allow the user to input information into system 700. The display 705 can also be used as an input device (e.g., a touchscreen display).
[0087] If included, optional side input elements allow for further user input. For example, a side input element could be a rotary switch, a button, or any other type of manual input element. Alternatively, system 700 may include more or fewer input elements. For example, in some embodiments, display 705 may not be a touchscreen. In another example, an optional keypad 735 may also be included, which could be a physical keypad or a “soft” keypad generated on a touchscreen display.
[0088] In various embodiments, output elements include a display 705 for displaying a graphical user interface (GUI), a visual indicator (e.g., a light-emitting diode 720), and / or an audio transducer 725 (e.g., a speaker). In some aspects, a vibration transducer is included to provide tactile feedback to the user. In another aspect, input and / or output ports, such as audio inputs (e.g., a microphone jack), audio outputs (e.g., a headphone jack), and video outputs (e.g., an HDMI port), are included for sending signals to or receiving signals from external devices.
[0089] One or more applications 766 may be loaded into memory 762 and run on or associated with operating system 764. Examples of applications include telephone dialers, email programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, conferencing programs, etc. System 700 also includes a non-volatile storage area 768 within memory 762. The non-volatile storage area 768 may be used to store persistent information that should not be lost if system 700 is powered off. Applications 766 may use and store information in the non-volatile storage area 768, such as emails or other messages used by email applications. A synchronization application (not shown) also resides on system 700 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep the information stored in the non-volatile storage area 768 synchronized with the corresponding information stored on the host computer. It should be understood that other applications may be loaded into memory 762 and run on system 700 as described herein.
[0090] System 700 has a power supply 770, which can be implemented as one or more batteries. The power supply 770 may also include an external power source, such as an AC adapter or a power dock for replenishing or recharging the battery.
[0091] System 700 may also include a radio interface layer 772, which performs the functions of transmitting and receiving radio frequency communications. Radio interface layer 772 facilitates wireless connectivity between system 700 and the "external world" through a communications operator or service provider. Transmissions to and from radio interface layer 772 are conducted under the control of operating system 764. In other words, communications received by radio interface layer 772 can be distributed to application 766 by operating system 764, and vice versa.
[0092] A visual indicator 720 can be used to provide visual notifications, and / or an audio interface 774 can be used to generate audible notifications via an audio transducer 725. In the illustrated embodiment, the visual indicator 720 is a light-emitting diode (LED), and the audio transducer 725 is a speaker. These devices can be directly coupled to a power supply 770 such that when activated, they remain on for the duration specified by the notification mechanism, even if the processor 760 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes an action to indicate the device's power-on status. The audio interface 774 is used to provide and receive audible signals to and from the user. For example, in addition to being coupled to the audio transducer 725, the audio interface 774 can also be coupled to a microphone to receive audible input, such as for telephone conversations. According to embodiments of this disclosure, the microphone can also be used as an audio sensor to control notifications, as will be described below. The system 700 may also include a video interface 776 that enables the onboard camera 730 to operate to record still images, video streams, etc.
[0093] It should be understood that system 700 may have additional features or functions. For example, system 700 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 7 The non-volatile storage area 768 is shown in the middle.
[0094] As described above, data / information generated, captured, and stored via system 700 can be stored locally, or the data can be stored on any number of storage media, which can be accessed by the device via radio interface layer 772 or via a wired connection between system 700 and a separate computing device associated with system 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via radio interface layer 772 or via a distributed computing network. Similarly, such data / information can be easily transferred between computing devices for storage and use, according to any of the various data / information transmission and storage means, including email and collaborative data / information sharing systems.
[0095] Figure 8 As described above, one aspect of the architecture of a system for processing data received at a computing system from a remote source (such as a personal computer 804, a tablet computing device 806, or a mobile computing device 808) is illustrated. Content displayed at server device 802 can be stored in different communication channels or other storage types. For example, various documents can be stored using a directory service 824, a web portal 825, an email service 826, instant messaging storage 828, or a social networking site 830.
[0096] Application 820 (e.g., similar to application 620) can be adopted by a client communicating with server device 802. Additionally or alternatively, application 821 can be adopted by server device 802. Server device 802 can provide data between server device 802 and client computing devices (such as personal computer 804, tablet computing device 806, and / or mobile computing device 808 (e.g., smartphone)) via network 815. As an example, the aforementioned computer system can be embodied in personal computer 804, tablet computing device 806, and / or mobile computing device 808 (e.g., smartphone). In addition to receiving graphics data that can be used for preprocessing at the graphics source system or post-processing at the receiving computing system, any of these computing device examples can also obtain content from repository 816.
[0097] It should be understood that the aspects and functions described herein can run on distributed systems (e.g., cloud-based computing systems), where application functions, memory, data storage and retrieval, and various processing functions can operate remotely to each other via distributed computing networks (such as the Internet or intranets). Various types of user interfaces and information can be displayed via onboard computing device displays or via remote display units associated with one or more computing devices. For example, various types of user interfaces and information can be displayed and interacted with on a wall surface, or projected onto that wall surface. Interaction with multiple computing systems on which embodiments of the present invention can be practiced includes keystroke input, touchscreen input, voice or other audio input, gesture input, etc., wherein the associated computing device is equipped with detection (e.g., camera) capabilities to capture and interpret user gestures to control the functions of the computing device.
[0098] In one aspect, a system is provided that includes at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations includes instantiating a digital agent having a role associated with a user, and receiving an instruction to assign the digital agent to a virtual meeting on behalf of the user. The set of operations also includes monitoring the virtual meeting, receiving first meeting input, and processing the first meeting input by a first base model to generate a first output. The set of operations includes determining a first action and determining a first rendering for the first action based on the first output. Additionally, the set of operations includes causing the digital agent to perform the first action on behalf of the user based on the role and the determined first rendering.
[0099] In a further aspect, the role is associated with a realistic representation of the user, a fantastical representation of the user, or any representation within that range. Additionally, monitoring can be performed continuously, at regular time intervals, or on a dynamic window. In a further aspect, the first rendering of the determined first action is an audio rendering, a visual rendering, or an audiovisual rendering. This set of operations also includes receiving a second conference input and processing the second conference input by a second base model to generate a second output. Based on the second output, a second action is determined and a second rendering of the second action is determined. Furthermore, the digital agent is made to perform the second action on behalf of the user based on the role and the determined second rendering. In a further aspect, the first base model is the same as the second base model or the first base model is different from the second base model. Additionally, the first rendering of the determined first action is at least partially based on the role. In a further aspect of the system, determining the first action includes one of the following: asking the user, querying the same or different base models, or selecting a pre-cached idea. This set of operations also includes: in response to the digital agent performing the first action, detecting the digital agent's proprioception and storing the proprioception.
[0100] On the other hand, a method is provided for instantiating a digital agent using one or more base models. The method includes instantiating a digital agent having a role associated with a user, wherein the role is associated at least with a voice simulation and a visual representation similar to the user, and receiving instructions to assign the digital agent to a virtual meeting on behalf of the user. Additionally, the method includes monitoring the virtual meeting, receiving meeting input, and processing the meeting input by the base models to generate model output. Based on the model output, the method includes determining an action, determining a rendering for the action, and causing the digital agent to perform the action on behalf of the user according to the role and the determined rendering.
[0101] Further aspects include using a state machine to monitor virtual meetings, where the state machine implements a thought loop. Additionally, the state machine returns to the monitoring after each iteration of the thought loop. The method also includes: detecting and storing the proprioception of the digital agent in response to the digital agent performing an action. Furthermore, the rendering of the determined action is at least partially based on the role.
[0102] In another aspect, a method is provided for instantiating a digital agent using one or more base models. The method includes instantiating a digital agent having a role associated with a user, and receiving an instruction to assign the digital agent to a virtual meeting on behalf of the user. Additionally, the method includes monitoring the virtual meeting by a state machine, receiving meeting input by the state machine, and generating prompts that include at least the meeting input. Based on the prompts, the method includes querying a first base model and receiving a first output from the first base model. Based on the first output, the method includes determining an action and causing the digital agent to perform the action on behalf of the user according to its role.
[0103] Additional aspects include updating cue words using actions and querying the second base model based on the updated cue words. Furthermore, the first base model may be the same as the second base model, or the first base model may be different from the second base model. Additionally, the state machine implements a think loop, which includes receiving state information from previous iterations, receiving conference input, requesting model output, receiving model output, processing model output, and sending state information to the next iteration. The method also includes determining the rendering of an action and having the digital agent execute the action based on that rendering.
[0104] For example, the aspects of this disclosure have been described above with reference to block diagrams and / or operational illustrations of methods, systems, and computer program products according to various aspects of this disclosure. The functions / actions marked in the boxes may not occur in the order shown in any flowchart. For example, two boxes shown consecutively may actually be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order, depending on the functions / actions involved.
[0105] The description and illustrations of one or more aspects provided in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to demonstrate mastery of the claimed aspects of the disclosure and to enable others to make and use those aspects. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Whether shown and described in combination or separately, various features (both structural and methodological features) are intended to be selectively included or omitted to produce embodiments with a particular set of features. Upon receiving the description and illustrations of this application, those skilled in the art can conceive of variations, modifications, and substitutions that fall within the spirit of the broader aspects of the general inventive concept embodied in this application and do not depart from the broader scope of the claimed disclosure.
Claims
1. A system comprising: At least one processor; as well as A memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations including: Instantiate (502) a digital agent, the digital agent having a role associated with the user; Receive (504) an instruction to assign the digital agent to a virtual meeting on behalf of the user; Monitoring (506) the virtual meeting; Receive (508) first conference input; The first conference input is processed by the first basic model (510) to generate the first output; Based on the first output, determine the first action (514, 516); Determine (516) the first rendering for the first action; and The digital agent (518) performs the first action on behalf of the user based on the role and the determined first rendering.
2. The system of claim 1, wherein the character is associated with a realistic representation of the user, a fantastical representation of the user, or any representation within the range of such representations.
3. The system according to claim 1, wherein the monitoring is performed continuously, at regular time intervals, or on a dynamic window.
4. The system of claim 1, wherein the first rendering of the determined first action is audio rendering, visual rendering, or audiovisual rendering.
5. The system according to claim 1, wherein the set of operations further includes: Receive input from the second conference; The second conference input is processed by the second basic model to generate the second output; Based on the second output, determine the second action; Determine the second rendering for the second action; as well as The digital agent, based on the role and the determined second rendering, performs the second action on behalf of the user.
6. The system according to claim 5, wherein the first basic model is the same as the second basic model or the first basic model is different from the second basic model.
7. The system of claim 1, wherein the first rendering of the determined first action is at least partially based on the character.
8. The system of claim 1, wherein determining the first action comprises one of: asking the user, querying the same or different base model, or selecting a pre-cached idea.
9. The system according to claim 1, wherein the set of operations further comprises: In response to the digital agent performing the first action, the proprioception of the digital agent is detected; as well as Store the proprioceptive sensation.
10. A method for instantiating a digital intelligent agent using one or more base models, comprising: Instantiate (502) a digital agent having a role associated with a user, wherein the role is associated at least with a voice simulation and a visual representation similar to the user; Receive (504) an instruction to assign the digital agent to a virtual meeting on behalf of the user; Monitoring (506) the virtual meeting; Receive (508) conference input; The conference input is processed by the base model (510) to generate the model output; Based on the model output, determine the (514, 516) action; Determine (516) the rendering for the action; as well as The digital agent (518) performs the action on behalf of the user based on the role and the determined rendering.
11. The method of claim 10, further comprising: A state machine is used to monitor the virtual meeting, wherein the state machine implements a think loop.
12. The method of claim 11, wherein the state machine returns to the monitoring after each iteration of the thinking loop.
13. The method of claim 10, further comprising: In response to the digital agent performing the action, the proprioceptive perception of the digital agent is detected; as well as Store the proprioceptive sensation.
14. A method for instantiating a digital intelligent agent using one or more base models, comprising: Instantiate (502) a digital agent, the digital agent having a role associated with the user; Receive (504) an instruction to assign the digital agent to a virtual meeting on behalf of the user; The virtual meeting is monitored by a state machine (506); The state machine receives the conference input (508); The generated (509) includes at least the prompt words input from the meeting; Based on the prompt words, query (511) the first basic model; Receive (512) the first output from the first basic model; Based on the first output, determine the action (514, 516); as well as The digital agent (518) performs the action on behalf of the user according to the role.
15. The method of claim 14, further comprising: Update the prompt word using the action; as well as The second base model is queried based on the updated prompts.
