Generating images for video communication sessions

Abstract

The media application obtains transcribed text from audio associated with the video communication session. The media application provides the transcribed text to a text generation machine learning model. The text generation machine learning model outputs a text cue based on the transcribed text, wherein the text cue includes an entity in the transcribed text. The media application provides the textual cues to the image generation machine learning model. The image generation machine learning model outputs a generated image in response to the text cue, wherein the generated image includes a depiction of the entity in the transcribed text. The media application causes the generated image to be displayed in the video communication session.

Description

Background Art

[0001] Generating images for a video conference that is occurring in real time is difficult because topics can be varied and can change quickly. A user can retrieve an image from a database and add the image to the video conference, but the time it takes to identify and retrieve the image may render the image irrelevant by the time the user locates and shares the image.

[0002] Furthermore, when a user saves a recording of a video communication session, it may be difficult for the user to remember the topics discussed in the video communication session. This problem may be exacerbated with each additional video communication that the user saves.

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors (to the extent that it is described in this background section) and aspects of the specification that might not have been considered prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present disclosure. Summary of the invention

[0004] A computer-implemented method includes obtaining transcribed text from audio associated with a video communication session. The method further includes providing the transcribed text to a text generation machine learning model. The method further includes outputting a text prompt based on the transcribed text using the text generation machine learning model, wherein the text prompt includes an entity in the transcribed text. The method further includes providing the text prompt to an image generation machine learning model. The method further includes outputting, using the image generation machine learning model, a generated image responsive to the text prompt, wherein the generated image includes a depiction of an entity in the transcribed text. The method further includes causing the generated image to be displayed in the video communication session.

[0005] In some embodiments, the method further includes identifying entities in the transcribed text by generating a summary of the transcribed text, and comparing the summary to a plurality of entity clusters to identify entities based on corresponding distances between the summary and the plurality of entity clusters, wherein the summary is provided to a text generation machine learning model. In some embodiments, the generated image is displayed as a background image behind a video of one or more participants in the video communication session.

[0006] In some embodiments, the method further includes: obtaining additional transcribed text from audio associated with the video communication session; outputting a further text prompt based on the additional transcribed text using a text generation machine learning model, wherein the further text prompt includes an additional entity in the additional transcribed text; providing the further text prompt to an image generation machine learning model; and updating a generated image in response to the further text prompt using the image generation machine learning model, wherein the updated generated image includes a depiction of the additional entity in the additional transcribed text. In some embodiments, the video communication session is a live session, and the method is performed multiple times during the live session using incremental audio received during periods between consecutive executions of the method.

[0007] In some embodiments, the entity includes a plurality of entities, and the plurality of entities are transformed into other entities based on the incremental audio. In some embodiments, the method further includes: generating a summary of the transcribed text; and indexing the summary of the transcribed text with a thumbnail version of the generated image. In some embodiments, the method further includes: scoring the set of entities based on visual aspects associated with each entity in the set of entities using a text generation machine learning model, wherein outputting a text prompt includes: outputting a text prompt with an entity associated with the highest score. In some embodiments, the entity is a plurality of entities, a first entity is based on audio associated with a video communication session from a first user, a second entity is based on audio associated with a video communication session from a second user, and the generated image depicts a logical connection between the first entity and the second entity.

[0008] In some embodiments, the method further comprises: receiving prewritten text prior to the video communication session; outputting one or more images based on entities detected in the prewritten text using an image generation machine learning model; detecting that the transcribed text matches a specific portion of the prewritten text; and causing the corresponding pregenerated image to be displayed in the video communication session. In some embodiments, the method further comprises: generating graphical data for displaying a user interface, the user interface comprising a set of suggested backgrounds for use during the video communication session, wherein the set of suggested backgrounds comprises one or more images. In some embodiments, the method further comprises: providing an option to save the generated image in association with the transcribed text of the video communication session. In some embodiments, the method further comprises: deleting the transcribed text after the video communication session ends.

[0009] In some embodiments, a non-transitory computer-readable medium having instructions stored thereon, which when executed by one or more processors causes the one or more processors to perform operations comprising: obtaining transcribed text from audio associated with a video communication session; providing the transcribed text to a text generation machine learning model; outputting a text prompt based on the transcribed text using the text generation machine learning model, wherein the text prompt includes an entity in the transcribed text; providing the text prompt to an image generation machine learning model; generating an image generated in response to the text prompt using the image generation machine learning model, wherein the generated image includes a depiction of the entity in the transcribed text; and causing the generated image to be displayed in the video communication session.

[0010] In some embodiments, the operation further includes: identifying entities in the transcribed text by: generating a summary of the transcribed text, and comparing the summary to a plurality of entity clusters to identify entities based on corresponding distances between the summary and the plurality of entity clusters, wherein the summary is provided to a text generation machine learning model. In some embodiments, the generated image is displayed as a background image behind a video of one or more participants in the video communication session. In some embodiments, the operation further includes: obtaining additional transcribed text from audio associated with the video communication session; outputting a further text prompt based on the additional transcribed text using the text generation machine learning model, wherein the further text prompt includes additional entities in the additional transcribed text; providing the further text prompt to the image generation machine learning model; and updating the generated image in response to the further text prompt using the image generation machine learning model, wherein the updated generated image includes a depiction of the additional entity in the additional transcribed text.

[0011] In some embodiments, a computing device includes: one or more processors; and a memory coupled to the one or more processors, the memory having instructions stored thereon, which when executed by the processors cause the processors to perform operations. The operations include: obtaining transcribed text from audio associated with a video communication session; providing the transcribed text to a text generation machine learning model; outputting a text prompt based on the transcribed text using the text generation machine learning model, wherein the text prompt includes an entity in the transcribed text; providing the text prompt to an image generation machine learning model; generating a generated image responsive to the text prompt using the image generation machine learning model, wherein the generated image includes a depiction of the entity in the transcribed text; and causing the generated image to be displayed in the video communication session.

[0012] In some embodiments, the operations further include: identifying entities in the transcribed text by: generating a transcript of the transcribed text, and comparing the transcript to a plurality of entity clusters to identify entities based on corresponding distances between the transcript and the plurality of entity clusters, wherein the transcript is provided to a text generation machine learning model. In some embodiments, the generated image is displayed as a background image behind a video of one or more participants in the video communication session. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram of an example network environment to generate images for a video communication session according to some embodiments described herein.

[0014] Figure 2 is a block diagram of an example computing device to generate images for a video communication session in accordance with some embodiments described herein.

[0015] Figure 3 An example user interface including a generated image according to some embodiments described herein is illustrated.

[0016] Figure 4 An example user interface including a background generated image according to some embodiments described herein is illustrated.

[0017] Figure 5 An example user interface with suggested contexts pre-generated for use during a video communication session is illustrated in accordance with some embodiments described herein.

[0018] Figure 6 An example user interface with thumbnail versions of generated images indexed according to a video communication session is illustrated in accordance with some embodiments described herein.

[0019] Figure 7 An example flow chart illustrating a method for outputting a generated image for a video communication session according to some embodiments described herein.

[0020] Figure 8 Another example flow chart of a method to output a generated image for a video communication session according to some embodiments described herein is illustrated. DETAILED DESCRIPTION

[0021] Generating images for video conferences occurring in real time is difficult because the topics of video conferences can be varied and can change rapidly. The methods, systems, and non-transitory computer-readable media described herein generate images during a live video communication session, wherein the images represent topics and conversations in the video communication session, are updated along with the audio in the session, and automatically provide relevant visual content. The described technology uses both a text generation machine learning model and an image generation machine learning model to automatically generate relevant images, which, for example, depict one or more entities discussed in audio and / or text exchanged between participants in the video communication session. The described technology provides technical advantages by reducing the computational costs incurred when one or more participants in the video communication session perform an image search, preview multiple images, and select a specific image for inclusion in the video communication session. The technology is also advantageous because image content related to the live discussion topic is generated and displayed substantially in real time, which is not feasible for current manual image recognition.

[0022] In some embodiments, the technology can be implemented to generate images in advance of a video communication session (e.g., for storytelling, demonstrations, etc., where some aspects of the content of the video communication session are known in advance), and based on matching the live audio and / or text of the session with the previously generated images (and / or associated text), the generated images are automatically revealed in the video communication session. In this way, the described technology also saves computing costs incurred during the video communication session by pre-caching relevant images, so that few or no computing resources are utilized by participants during the video communication session to perform searches, image previews, or image selections.

[0023] In some embodiments, the text generation machine learning model transcribes text from the video communication session and outputs a text prompt including an entity that is included in the transcribed text. For example, a first user might discuss her activities in Central Park last weekend, and a second user might discuss that he recently saw an animated movie. The text generation machine learning model can output a text prompt including the name of Central Park and the animated movie.

[0024] In some embodiments, the image generation machine learning model receives a text prompt and outputs a generated image that includes a depiction of one or more entities in the transcribed text. For example, the image generation machine learning model may output a generated image that includes a depiction of Central Park, as if the generated image were in an animated film. An initial image of Central Park may be generated based on the audio of the first user, and may be updated to modify the depiction of Central Park to a visual style that matches the visual style of the animated film. The generated image may also be used as a thumbnail image that is used to index the minutes of the transcribed text for future retrieval. This may improve the ability to query a data repository on which the transcribed text (or other associated content, such as an audio / video recording) is stored, thereby improving efficiency, for example, by allowing the query process to operate on visual elements rather than dense text transcriptions. Users may more quickly and easily identify visual elements than text, and / or the search query process may provide results based on the characteristics of the thumbnail image.

[0025] Sample Environment 100

[0026] Figure 1 A block diagram of an example environment 100 for generating images for a video communication session is illustrated. In some embodiments, the environment 100 includes a media server 101 coupled to a network 105, a user device 115a, and a user device 115n. Users 125a, 125n may be associated with respective user devices 115a, 115n. In some embodiments, the environment 100 may include Figure 1 Other servers or devices not shown in FIG. Figure 1 In the remaining figures, a letter following a reference number (e.g., "115a") indicates a reference to the element having that particular reference number. A reference number in the text without a following letter (e.g., "115") indicates a general reference to the embodiment of the element carrying that reference number.

[0027] The media server 101 may include a processor, a memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection such as Ethernet, a coaxial cable, a fiber optic cable, etc., or a wireless connection such as Wi-Fi®, Bluetooth® or other wireless technologies. In some embodiments, the media server 101 sends data to one or more of the user devices 115a, 115n via the network 105 and receives data from the one or more. The media server 101 may include a media application 103a and a database 199.

[0028] Database 199 may store machine learning models, training data sets, video communication sessions (with user permission), generated images (with user permission), etc. Database 199 may also store social network data associated with user 125, user preferences of user 125, etc.

[0029] User device 115 may be a computing device that includes a memory coupled to a hardware processor. For example, user device 115 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing network 105.

[0030] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored on user device 115a as media application 103b and / or stored on user device 115n as media application 103c. Signal lines 108 and 110 may be wired connections such as Ethernet, coaxial cable, fiber optic cable, etc., or wireless connections such as Wi-Fi®, Bluetooth®, or other wireless technologies. User devices 115a, 115n are accessed by users 125a, 125n, respectively. Figure 1 The user devices 115a and 115n in FIG. 1 are used as examples only. Figure 1 Two user devices 115 a and 115 n are illustrated, but the present disclosure is applicable to a system architecture having one or more user devices 115 .

[0031] In some embodiments, the operations described herein are performed on the media server 101 and / or the user device 115. In some embodiments, some operations may be performed on the media server 101 and some operations may be performed on the user device 115.

[0032] The execution of the operation is consistent with the user settings. For example, the user 125a can specify the setting that the operation is to be performed on its corresponding user device 115a instead of on the media server 101. In the case of having such settings, the operation described herein is performed completely on the user device 115a, and no operation is performed on the media server 101. Further, the user 125a can specify that the video and / or other data of the user are only locally stored on the user device 115a, and not stored on the media server 101. In the case of having such settings, no user data is sent to the media server 101 or stored on the media server. Only when the user has agreed to the sending, storage and execution of the operation by the media server 101, the sending of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and the execution of the operation of such data by the media server 101 can be performed. The user is provided with the option to change the setting at any time, for example, so that they can enable or disable the use of the media server 101.

[0033] Machine learning models (e.g., neural networks or other types of models) are stored and utilized locally on the user device 115 with specific user permissions when utilized for one or more operations. Server-side models are used only with permission by the user. Further, trained models may be provided for use on the user device 115. During such use, with permission by the user 125, on-device training of the model may be performed. Updated model parameters may be sent to the media server 101 with permission by the user 125, for example, to enable federated learning. The model parameters do not include any user data.

[0034] In some embodiments, media application 103 can be implemented using hardware, including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a can be implemented using a combination of hardware and software.

[0035] The media application 103 obtains transcribed text from the audio associated with the video communication session. In some embodiments, the media application 103 includes a text generation machine learning model that receives the transcribed text and outputs a text prompt based on the transcribed text. The text prompt includes entities in the transcribed text, such as locations, people, video games, etc.

[0036] The media application 103 includes an image generation machine learning model that receives a text prompt. The image generation machine learning model generates a generated image in response to the text prompt. The generated image includes a depiction of an entity in the transcribed text.

[0037] Example computing device 200

[0038] Figure 2 is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 for implementing a media application 103a. In another example, the computing device 200 is a user device 115.

[0039] In some embodiments, computing device 200 includes processor 235, memory 237, input / output (I / O) interface 239, microphone 241, speaker 243, display 245, camera 247, and storage device 249, all coupled via bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, microphone 241 may be coupled to bus 218 via signal line 228, speaker 243 may be coupled to bus 218 via signal line 230, display 245 may be coupled to bus 218 via signal line 232, camera 247 may be coupled to bus 218 via signal line 234, and storage device 249 may be coupled to bus 218 via signal line 236.

[0040] Processor 235 may be one or more processors and / or processing circuits to execute program code and control the basic operations of computing device 200. "Processor" includes any suitable hardware system, mechanism or component that processes data, signals or other information. The processor may include a system having a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core or multi-core configuration), multiple processing units (e.g., in a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for implementing functionality, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor or other system optimized for matrix calculations (e.g., matrix multiplication). In some embodiments, processor 235 may include one or more coprocessors that implement neural network processing. In some embodiments, processor 235 may be a processor that processes data to generate a probabilistic output, for example, the output generated by processor 235 may be inaccurate or may be accurate within a range from the expected output. Processing does not need to be limited to a specific geographic location or have a time limit. For example, a processor may perform the functions of the processor in real time, offline, in batch mode, etc. Portions of the processing may be performed at different times and different locations by different (or the same) processing systems.A computer may be any processor in communication with a memory.

[0041] The memory 237 is typically provided in the computing device 200 for access by the processor 235 and may be any suitable processor-readable storage medium (such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc.) suitable for storing instructions for execution by the processor or set of processors and located separately from and / or integrated with the processor 235. The memory 237 may store software operated by the processor 235 on the computing device 200, including the media application 103.

[0042] The memory 237 may include an operating system 262, other applications 264, and application data 266. The other applications 264 may include, for example, a video library application, a video management application, a video zone application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in a number of environments and platforms, such as as a stand-alone computer program that can run on any type of computing device, as a web application with a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0043] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include videos used by a video library application, user actions recognized by other applications 264 (e.g., a social networking application), and the like.

[0044] The I / O interface 239 may provide functionality to enable the computing device 200 to interface with other systems and devices. The interface device may be included as part of the computing device 200, or may be separate and communicate with the computing device 200. For example, a network communication device, a storage device (e.g., memory 237 and / or storage device 249), and an input / output device may communicate via the I / O interface 239. In some embodiments, the I / O interface 239 may be connected to an interface device such as an input device (keyboard, pointing device, touch screen, microphone, scanner, sensor, etc.), and / or an output device (display device, speaker device, printer, monitor, etc.).

[0045] Microphone 241 may include hardware for detecting sound. For example, microphone 241 may detect ambient noise, human speech, music, etc. using a single microphone 241 that is part of user device 115.

[0046] In some embodiments, microphone 241 includes additional hardware for processing the audio captured when the user is recording a video. An analog-to-digital converter can convert an analog electrical signal into a digital electrical signal. A digital signal processor can convert the digital electrical signal into a digital output signal that is sent to speaker 243.

[0047] Speaker 243 may include hardware for producing audio signals heard by a user. In some embodiments, speaker 243 includes an amplifier for amplifying certain channels, frequencies, etc.

[0048] Display 245 includes hardware for displaying content (e.g., images, videos, and / or user interfaces of output applications as described herein) and for receiving touch (or gesture) input from a user. For example, display 245 may be utilized to display a user interface that includes a set of suggested backgrounds for use during a video communication session. Display 245 may include any suitable display device, such as a liquid crystal display (LCD), light emitting diode (LED), or plasma display screen, cathode ray tube (CRT), television, monitor, touch screen, three-dimensional display screen, or other visual display device. For example, display 245 may be a flat display screen disposed on a mobile device, multiple display screens embedded in an eyeglass form factor or head mounted device, or a monitor screen for a computer device.

[0049] Camera 247 can be any type of image capture device capable of capturing images and / or video. In some embodiments, camera 247 captures images or videos that I / O interface 239 sends to media application 103.

[0050] The storage device 249 stores data related to the media application 103. For example, the storage device 249 can store training data sets, text generation machine learning models, image generation machine learning models, videos (with user permission), minutes (with user permission), etc.

[0051] Figure 2 An example media application 103 is illustrated, which includes a video module 202, a text generation module 204, an image generation module 206, and an indexer 208. In some embodiments, each of the modules includes a set of instructions executable by the processor 235 to perform the steps discussed in more detail below. In some embodiments, each of the components is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0052] Various embodiments described herein may include program analysis of audio, video, text, or other media as part of a video communication session. For example, audio may include verbal audio (or other audio, such as recorded audio, audio detected by a participant's microphone, etc.) from a participant in a video communication session, video may include (e.g., from a camera) a close-up video of a participant or other video content (e.g., a shared screen, streaming video, etc.), text may include chat messages exchanged between participants, and other media may include files (e.g., documents, images, multimedia objects, etc.), etc. Program analysis of audio, video, text, or other media (content) is performed with specific user permission from a participant in the video communication session (e.g., a participant providing specific content, a participant receiving specific content, all participants in a video communication session, a host or sponsor of a video communication session, etc.). Participants are provided with notification that such program analysis can be performed and can choose to selectively enable or disable program analysis. If a participant refuses permission, no program analysis is performed. Further, the analyzed content is processed in accordance with applicable laws and regulations, and is processed in a secure manner (e.g., locally on a user device, or centrally on a server, using encryption and / or other security technologies). Without user permission, no content is stored. Further, for certain sets of users (e.g., users who do not meet age criteria, users associated with a particular organization where organizational policy prevents program analysis, etc.), these techniques are completely disabled.

[0053] The video module 202 facilitates a video communication session. For example, the video module 202 may be stored on a server and include instructions to receive a first video stream from a first user device and send the first video stream to a second user device. The video stream includes audio. In some embodiments, the video module 202 transcribes the audio into transcribed text. For example, the video module 202 may include a transcription machine learning model or other speech to text engine.

[0054] The video module 202 obtains transcribed text from the audio associated with the video communication session. The video module 202 can send the transcribed text to the text generation module 204.

[0055] In some embodiments, the video module 202 obtains additional transcribed text from the audio associated with the video communication session. For example, the video communication session can be a live session, and the video module 202 can generate transcribed text using incremental audio when additional audio is received during the video communication session. In some embodiments, the video module 202 generates transcribed text iteratively (such as after each person finishes speaking a word or a sentence, every minute, every five minutes, etc.). The video module 202 can send the additional transcribed text to the text generation module 204 as the transcribed text is generated.

[0056] In some embodiments, the text generation module 204 generates a summary based on the transcribed text. The summary may include a list of participants, entities discussed in the transcribed text (e.g., Sarah went to the Natural History Museum next to Central Park on Sunday), sentiments associated with the entities (e.g., Sarah had the best time), etc. In this example, the entities may include "Sarah", "Natural History Museum", "Central Park", and "Sunday", and the sentiments may include "delight", "happiness", etc. (associated with the text "had the best time").

[0057] In some embodiments, the text generation module 204 includes a text generation machine learning model that receives the transcribed text as input and outputs the minutes. The text generation machine learning model can be a large language model (LLM). In some embodiments, the text generation module 204 compares the minutes with multiple entity clusters to identify one or more entities in the minutes based on corresponding distances between the minutes and the multiple entity clusters. In some embodiments, the text generation module 204 supplements the minutes with a knowledge graph that includes information about the entities. The knowledge graph can be part of the media application 103 or part of a third-party service. Instead of the transcribed text, the minutes can be provided to the text generation machine learning model.

[0058] In some embodiments, the text generation module 204 uses a text generation machine learning model to output a text prompt based on the transcribed text, or based on the minutes if the text generation module 204 also includes the minutes. The text prompt includes one or more entities in the transcribed text. Continuing with the above example, the text generation machine learning model may receive a minute and / or transcribed text describing that Sarah went to the museum on Sunday and had the best time. The text generation machine learning model may output a text prompt requesting an image of an older museum building made of bricks, which is located next to an overgrown garden where ivy has invaded the bricks of the museum. The text prompt may request an older museum building based on a "Natural History Museum." Conversely, if the museum is a modern art museum, the text prompt may include "art styles from the 20th or 21st century."

[0059] In some embodiments, the text generation machine learning model is trained by the text generation module 204 and may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep learning neural network that implements multiple layers (e.g., a "hidden layer" located between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, uses one or more neural network layers to process each tile separately and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that receives sequence data (such as words in a sentence, frames in a video, etc.) as input and produces a sequence of results as output), etc.

[0060] The model form or structure can specify the connectivity between the various nodes and the organization of the nodes into layers. For example, the nodes of the first layer (e.g., the input layer) can receive transcribed text as input data or application data 266. Such data can include, for example, one or more words or phrases per node when the trained model is used for analysis of, for example, text. Subsequent intermediate layers can receive the output of the nodes of the previous layer as input according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. The last layer (e.g., the output layer) produces the output of the machine learning model, such as a text prompt. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0061] In some embodiments, the text generation machine learning model identifies a set of entities in the transcribed text and outputs a score associated with each entity. In some embodiments, the text generation machine learning model outputs higher scores for more visual entities than for fewer visual entities. For example, a higher score may be associated with the entity "sunflower" in "I saw sunflowers" compared to the score associated with the entity "music" in "I heard good music". If two types of entities appear close to each other, such as "I heard good music in a cafe", the entity "cafe" may be associated with a higher score than the entity "music". In some embodiments, newer entities discussed in the transcribed text are given higher priority than older entities discussed in the transcribed text. The text generation machine learning model can rate the entity set based on the corresponding scores and output a text prompt with the entity associated with the highest score. In some embodiments, scoring is performed by an intermediate layer in the neural network.

[0062] In some embodiments, the text generation module 204 may include multiple trained text generation machine learning models. One or more of the text generation machine learning models may include multiple nodes arranged in layers according to the model structure or form. In some embodiments, the node may be, for example, a computing node without memory configured to process an input unit to generate an output unit. The calculation performed by the node may include, for example, multiplying each node input in the multiple node inputs by a weight to obtain a weighted sum, and adjusting the weighted sum using a bias or intercept value to generate a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, the calculations performed by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a graphics processing unit (GPU), or a dedicated neural circuit. In some embodiments, the node may include a memory, for example, it may be possible to store and use one or more earlier inputs when processing subsequent inputs. For example, a node with a memory may include a long short-term memory (LSTM) node. LSTM nodes may use memory to maintain a "state" that allows the node to function like a finite state machine (FSM).

[0063] In some embodiments, the trained model may include embeddings or weights for each node. For example, the model may be initiated as a plurality of nodes organized into layers as specified by the model form or structure. Upon initialization, corresponding weights may be applied to the connections between each pair of nodes connected in the model form (e.g., nodes in successive layers of a neural network). For example, corresponding weights may be randomly assigned or initialized to default values. The text generation machine learning model may then be trained, for example using the training data, to produce results.

[0064] The training can be performed using supervised learning techniques. In supervised learning, the training data may include multiple inputs (e.g., multiple transcribed text documents) and corresponding ground truth outputs for each input (e.g., text prompts for each transcribed text document). Based on a comparison of the output of the model (e.g., predicted text prompts) with the ground truth outputs (e.g., ground truth minutes), the values ​​of the weights are automatically adjusted, for example, in a manner that increases the probability that the model produces the ground truth passage.

[0065] In some embodiments, the training is unsupervised.The text can be divided into a plurality of clusters, and the clusters can be organized according to the similarity of the text.

[0066] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained text generation machine learning model may include, for example, an initial set of weights downloaded from a server that provides weights. In various embodiments, the trained text generation machine learning model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the text generation module 204 may generate a trained text generation machine learning model based on a priori training performed, for example, by a developer of the text generation module 204, by a third party, etc.

[0067] In some embodiments, where the text generation machine learning model includes a convolutional neural network trained using supervised learning, the training of the text generation machine learning model may include, for each training transcribed text, obtaining text prompts based on the transcribed text. The text generation machine learning model may calculate a loss value based on a comparison of the predicted text prompts for the transcribed text with a benchmark true text prompt (included in the training data). The text generation machine learning model may update the weights of one or more nodes of the convolutional neural network based on the loss value (e.g., in such a way that after adjusting and running another training cycle, the loss value is reduced until the loss value is below a threshold). In some embodiments, the text generation machine learning model includes a learnable convolutional encoder and decoder layer and a time-domain convolutional network masking network.

[0068] Once the text generation machine learning model is trained, the text generation machine learning model receives the transcribed text (from the video communication session) as input and outputs a text prompt. The text generation module 204 provides the text prompt as input to the image generation module 206.

[0069] In some embodiments, the text generation machine learning model receives additional transcribed text from the video module 202 and generates further text prompts based on the additional transcribed text.

[0070] The image generation module 206 may include an image generation machine learning model that receives a text prompt as input and outputs a generated image. The generated image is responsive to the text prompt and includes a depiction of an entity in the transcribed text.

[0071] In some embodiments, the image generation module 206 trains an image generation machine learning model using training data including textual prompts as input and generates images as baseline real data.

[0072] The image generation machine learning model can be an autoregressive text-to-image generation model that generates images that support context-rich synthesis involving complex compositions and world knowledge. In some embodiments, the image generation machine learning model encodes the image as a sequence of discrete tokens.

[0073] Alternatively, the image generation machine learning model can use a diffusion model to output the generated image. The diffusion model can perform text conditioning on the text prompt. For example, if the text request is for replacing the shirt that the subject is wearing in the initial image with a blue shirt, the diffusion model performs text conditioning by generating a blue shirt.

[0074] The diffusion model can perform a diffusion process on a noisy image. The diffusion model is trained by adding noise to the image and training the diffusion model to remove the noise via a denoising process. During actual use, the diffusion model applies the denoising process to a random seed to generate a realistic image. The diffusion model generates a noisy image by simulating diffusion, and then performs inverse diffusion, which is the process by which the output image emerges from the noise.

[0075] In some embodiments, the diffusion model first performs inverse diffusion to create a noisy image, provides the noisy image to a convolutional neural network with a self-attention mechanism for performing feature extraction, and then performs forward diffusion, which combines the noisy image with textual conditioning to generate an output image that satisfies the textual cues provided as input to the diffusion model. In some embodiments, the diffusion model performs inverse diffusion using a denoising diffusion implicit model (DDIM) inversion.

[0076] The trained image generation machine learning model receives a text prompt from the text generation machine learning model. The image generation machine learning model outputs a generated image that is responsive to the text prompt and includes a depiction of an entity in the transcribed text.

[0077] In some embodiments, the image generation machine learning model receives further text prompts and updates the generated image using the image generation machine learning model in response to the further text prompts. The image generation machine learning model can gradually update the generated image over time while retaining the depiction of each entity in the initially generated image. For example, a first note may include "There is a planet called Earth," and the generated image is Earth. A second note may include "Dinosaurs live on this planet," and the updated generated image includes Earth with dinosaurs on its surface. A third note may include "Earth was hit by a meteorite," and the updated generated image includes Earth with dinosaurs being hit by a meteorite. A fourth note may include "Dinosaurs are extinct and Earth turned into a fireball," wherein the updated generated image includes a fireball.

[0078] Once the video communication session has ended, the text generation machine learning model can generate a summary of the transcribed text. The summary can include a merger of previous minutes iteratively generated during the live video communication session, or the summary can be generated based on the complete transcribed text representing the entire video communication session. In some embodiments, the indexer 208 can use the generated image to index the summary of the transcribed text. For example, the summary of the transcribed text can be indexed with a thumbnail version of the generated image.

[0079] In some embodiments, where the generated image is updated over time, the image generation module 206 may generate a video clip of the generated image and update the generated image that the indexer 208 indexes as a thumbnail version of the video clip associated with the minute. The thumbnail version of the generated image or video clip advantageously allows a user to quickly identify a particular video communication session based on viewing the thumbnail version.

[0080] The indexer 208 saves the minutes and corresponding thumbnail versions of the generated images with certain user permission. In some embodiments, the minutes are discarded after the video communication session is completed unless the user provides permission to save the minutes. In some embodiments, the indexer 208 provides the user with an option to save the generated images in association with the transcribed text of the video communication session, and indexes the generated images in response to receiving a selection of the option from the user. In some embodiments, if the user does not provide permission to save the minutes, the indexer 208 deletes the transcribed text and / or minutes after the video communication session ends. In some embodiments, if the user does not provide permission to save the minutes, the image generation module 206 may generate any generated images.

[0081] In addition to or as an alternative to using the generated images to index the above-mentioned minutes, the indexer may also be used to index other content that may be stored related to the video communication session (such as audio and / or visual recordings of the video communication session). In fact, generating images in this way to index files may not be limited to video communication sessions, and may also be applicable to, for example, audio communication sessions without any visual elements. However, it may be useful that the generated images have been displayed to the user during the communication session so as to facilitate identification of subsequent retrieval of the indexed content.

[0082] Sample User Interface

[0083] In some embodiments, the transcribed text describes multiple entities. For example, a first user may describe that on their last vacation, they took a hot air balloon ride for the first time. A second user may describe that on their last vacation, they went cross-country skiing in the mountains. In this example, the transcribed text includes the following entities: hot air balloon and the mountains where the person will go cross-country skiing.

[0084] The text generation module 204 receives the transcribed text and outputs a text prompt requesting a generated image including a hot air balloon and a mountain range where the personnel will go cross-country skiing. The image generation module 206 receives the text prompt and, based on the text prompt, outputs a generated image including the entities in the text prompt. In some embodiments, the generated image depicts a logical connection between the first entity and the second entity. For example, instead of including a generated image that includes a hot air balloon that is the same size as the mountain range, the generated image includes the hot air balloon that is sized and positioned so that the hot air balloon and the mountain range are part of the same scene.

[0085] In some embodiments, the generated image is displayed in the video communication session. For example, the generated image is displayed when the user hears the audio associated with the video communication session.

[0086] Figure 3 An example user interface 300 is illustrated that includes a generated image 307. The user interface 300 includes a video screen 305 with a generated image and a video communication session icon (such as a phone icon) that ends the video communication session when selected. The generated image 307 includes a hot air balloon 315 and a mountain range 320 where the persons are going to cross-country ski. Thus, the generated image advantageously combines visual aspects related to entities from each person participating in the video communication session.

[0087] As the video communication session continues, the video module 202 obtains additional transcribed text from the audio associated with the video communication session. The text generation machine learning model receives the additional transcribed text as input and outputs further text prompts. The image generation machine learning model receives the additional text prompts and updates the generated image to be responsive to the further text prompts. For example, continuing with the above details, the first user may additionally describe how they took a wine tasting tour after the hot air balloon trip.

[0088] The image generation machine learning model can update the generated image to include entities associated with wine tasting, such as rolling hills in Napa Valley, including the vineyards where they conducted the wine tasting event. The image generation module 206 can update the generated image to show one or more of the entities transformed into additional entities. For example, the mountains can be transformed into rolling hills in Napa Valley.

[0089] In some embodiments, the generated image is displayed as a background image behind the video of one or more participants in the video communication session. The image generation module 206 can output the generated image with negative space for the user's image to be placed. In some embodiments, the image generation module 206 adjusts the brightness, contrast, and color of the user's video stream so that it appears that the user is in the environment.

[0090] Figure 4 An example user interface 400 including a background generated image is illustrated. In this example, a first user 410 describes that during last weekend, he watched an animated film that took place primarily underwater. A second user 415 describes how she spent a day walking in Central Park over the weekend. The image generation machine learning model outputs a generated image 405 depicting a portion of Central Park, and underwater elements are added to the generated image.

[0091] In some embodiments, the generated image is displayed separately from the video of each user.For example, a video conference session may include a first image of a first user, a second image of a second user, and a third image that is a generated image.

[0092] In some embodiments, instead of a text prompt, an image generation machine learning model receives pre-written text as input and outputs one or more images based on entities detected in the pre-written text. The one or more images can be used as a recommended background set for use during a video communication session. For example, a parent may want to read a story to the parent's son using a video communication session, and the background set can include typical characters in the story, such as princesses, princes, castles, monsters, knights, etc. Because the image generation machine learning model can update the generated images based on the attached transcribed text, the suggested backgrounds serve as a useful starting point for the story, where entities transform into other entities as the parent reads the story.

[0093] Figure 5 An example user interface 500 with a suggested background pre-generated for use during a video communication session is illustrated. In this example, the image generation machine learning model outputs three images based on pre-written text 505, 510, 515.

[0094] In some embodiments, the image generation machine learning model can output an image based on user input. For example, the user can select Figure 5 Box 520 in provides an entity that is used directly by the image generation machine learning model to output a generated image, or that is received as input by the text generation machine learning model and used to generate a text prompt that is used by the image generation machine learning model to output a generated image.

[0095] In some embodiments, prior to the video communication session, the text generation machine learning model receives prewritten text and outputs a text prompt for the prewritten text. The image generation machine learning model may receive a text prompt for the prewritten text and output one or more images based on entities detected in the text prompt. Alternatively, the image generation machine learning model may directly receive the prewritten text and output one or more images based on entities detected in the prewritten text.

[0096] During the video communication session, the text generation machine learning model can detect that the transcribed text matches a particular portion of the pre-written text, and the image generation machine learning model can cause the corresponding pre-generated image to be displayed.

[0097] Figure 6 An example user interface 600 is illustrated with thumbnail versions of generated images 605, 610, 615, 620 indexed according to a video communication session. In some embodiments, selecting one of the thumbnails causes the corresponding minutes to be displayed.

[0098] Example Flowchart

[0099] Figure 7 An example flow chart of a method 700 for outputting a generated image for a video communication session is illustrated. The method 700 may be performed by Figure 2 In some embodiments, the method 700 is performed by the computing device 200 in Figure 1 The method may be executed on the user device 115 , the media server 101 , or partially on the user device 115 and partially on the media server 101 .

[0100] Figure 7 Method 700 may begin at block 702. At block 702, a determination is made as to whether permission is received from a user to access user data. If permission is not received, block 702 may be followed by block 704. At block 704, a notification is caused to be displayed that denies provision of the generated image. If permission is received, block 702 may be followed by block 706.

[0101] At block 706 , transcribed text is obtained from the audio associated with the video communication session. Block 706 may be followed by block 708 .

[0102] At block 708, the transcribed text is provided to a text generation machine learning model. Block 708 may be followed by block 710.

[0103] At box 710, the text generation machine learning model outputs a text prompt based on the transcribed text, wherein the text prompt includes an entity in the transcribed text. In some embodiments, the text generation machine learning model identifies the entity in the transcribed text by generating a summary of the transcribed text, and comparing the summary to a plurality of entities to identify the entity based on corresponding distances between the summary and a plurality of entity clusters. In some embodiments, the transcribed text includes a plurality of entities, and the method further includes scoring the set of entities based on visual aspects associated with each entity in the set of entities, wherein outputting the text prompt includes outputting a text prompt having an entity associated with the highest score. In some embodiments, the entity is a plurality of entities, a first entity is based on audio associated with a video communication session from a first user, a second entity is based on audio associated with a video communication session from a second user, and the generated image depicts a logical connection between the first entity and the second entity. Box 710 may be followed by box 712.

[0104] At block 712, the textual prompt is provided to the image generation machine learning model. Block 712 may be followed by block 714.

[0105] At block 714, the image generation machine learning model outputs a generated image in response to the text prompt, wherein the generated image includes a depiction of an entity in the transcribed text. Block 714 may be followed by block 716.

[0106] At box 716, the generated image is caused to be displayed in the video communication session. In some embodiments, the generated image is displayed as a background image behind the video of one or more participants in the video communication session. In some embodiments, the method further includes: providing an option to save the generated image in association with the transcribed text of the video communication session (or other content related to the video communication session, such as audio and / or visual recordings). In some embodiments, the thumbnail version of the generated image is used to index the minutes of the transcribed text and / or other content related to the video communication session. In some embodiments, the transcribed text is deleted after the video communication session ends.

[0107] In some embodiments, the method further includes: obtaining additional transcribed text from audio associated with the video communication session. The text generation machine learning model outputs a further text prompt based on the additional transcribed text, wherein the further text prompt includes additional entities in the additional transcribed text. In response to the further text prompt, the further text prompt is provided to the image generation machine learning model, and the updated generated image includes a depiction of the additional entity in the additional transcribed text.

[0108] In some embodiments, the video communication session is a live session, and the method is performed multiple times during the live session using incremental audio received during a period between consecutive executions of the method. In addition, instead of one entity, multiple entities are identified, and one or more of the multiple entities are transformed into one or more of the multiple entities based on the incremental audio.

[0109] In some embodiments, prior to the video communication session, the method further comprises: receiving prewritten text, the image generation machine learning model outputting one or more entities based on the entities detected in the prewritten text; detecting that the transcribed text matches a specific portion of the prewritten text; and causing the corresponding pregenerated image to be displayed. The method may further comprise: generating graphical data for displaying a user interface, the user interface comprising a set of suggested backgrounds for use during the video communication session.

[0110] Figure 8 Another example flow chart of a method 800 for outputting a generated image for a video communication session is illustrated. The method 800 may be performed by Figure 2 In some embodiments, the method 800 is performed by the computing device 200. Figure 1 The method may be executed on the user device 115 , the media server 101 , or partially on the user device 115 and partially on the media server 101 .

[0111] Figure 8Method 800 of the embodiment of the present invention may begin at block 802. At block 802, it is determined whether permission is received from the user to access user data. If permission is not received, block 802 may be followed by block 804. At block 804, a notification is displayed that denies provision of the generated image. If permission is received, block 802 may be followed by block 806.

[0112] At block 806, transcribed text is obtained from the audio associated with the video communication session. Block 806 may be followed by block 808.

[0113] At block 808, the transcribed text is provided to a first layer of a text generation machine learning model. Block 808 may be followed by block 810.

[0114] At block 810 , the text generation machine learning model outputs a note based on the transcribed text, wherein the note includes entities in the transcribed text. Block 810 may be followed by block 812 .

[0115] At block 812, the minutes are provided to a second layer of a text generation machine learning model. Block 812 may be followed by block 814.

[0116] At block 814, the text generation machine learning model outputs a text prompt based on the minutes. Block 814 may be followed by block 816.

[0117] At block 816, the textual hint is provided to the image generation machine learning model. Block 816 may be followed by block 818.

[0118] At block 818, the image generation machine learning model outputs a generated image in response to the text prompt, wherein the generated image includes a depiction of the entity in the transcribed text. Block 818 may be followed by block 820.

[0119] At block 820, the generated image is caused to be displayed in the video communication session. Block 820 may be followed by block 822.

[0120] At block 822, in response to ending the video communication session, the transcribed text and minutes are deleted.

[0121] Continuing with the above description, controls may be provided to the user, allowing the user to make choices regarding whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., use of video communication session data, generation of transcribed text, generation of minutes, generation of generated images, use of generative artificial intelligence, storage of data, etc., information about the user's activities, occupation, preferences of the user, or the user's current location), and whether to send content or communications from a server to the user. For a video communication session, all participants in the video communication session provide permission to use the previously mentioned data. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user may control what information is collected about the user, how the information is used, and what information is provided to the user.

[0122] In the above description, many specific details are set forth for the purpose of explanation in order to provide a thorough understanding of this specification. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details. In some cases, structures and devices are shown in block diagram form in order to avoid ambiguous descriptions. For example, the embodiments are described above mainly with reference to user interfaces and specific hardware. However, the embodiments can be applied to any type of computing device that can receive data and commands, and any peripheral device that provides services.

[0123] References to "some embodiments" or "some examples" in the specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiments or examples may be included in at least one implementation of the specification. The phrase "in some embodiments" appearing in various places in this specification does not necessarily refer to the same embodiment.

[0124] Some parts of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits in computer memory. These algorithmic descriptions and representations are means used by those skilled in the art of data processing to most effectively communicate the substance of their work to other technicians in the field. Here and generally contemplated algorithms are considered to be self-consistent sequences of steps leading to a desired result. The steps are steps that require physical manipulation of physical quantities. Although not necessary, these quantities are generally in the form of electrical or magnetic data that can be stored, transferred, combined, compared, and otherwise manipulated. It has been confirmed that, primarily for common usage reasons, these data can sometimes be appropriately referred to as bits, values, elements, symbols, characters, terms, numbers, etc.

[0125] It should be borne in mind, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically noted, as will be apparent from the following discussion, it should be understood that throughout the description, discussions utilizing terms including "processing" or "computing" or "calculating" or "determining" or "displaying" and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and converts that data into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0126] Embodiments of the present specification may also relate to a processor for performing one or more steps of the above method. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, which includes but is not limited to: any type of disk (including optical disk), ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory (including USB key with non-volatile memory), or any type of medium suitable for storing electronic instructions, each medium coupled to a computer system bus.

[0127] The specification may take the form of some complete hardware embodiments, some complete software embodiments, or some embodiments comprising both hardware elements and software elements. In some embodiments, the specification is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0128] Furthermore, the description may take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device.

[0129] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to a memory element via a system bus. The memory element may include local memory, mass storage devices, and cache memories employed during actual execution of the program code, which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from mass storage devices during execution.

Claims

1. A computer-implemented method comprising: obtaining transcribed text from audio associated with the video communication session; providing the transcribed text to a text generation machine learning model; outputting a text prompt based on the transcribed text using the text generation machine learning model, wherein the text prompt includes entities in the transcribed text; providing the text prompt to an image generation machine learning model; generating, using the image-generating machine learning model, an output of a generated image responsive to the text prompt, wherein the generated image includes a depiction of the entity in the transcribed text; and The generated image is caused to be displayed in the video communications session.

2. The method of claim 1, further comprising identifying the entity in the transcribed text by: generating a transcript of the transcribed text; and The minutes are compared to a plurality of entity clusters to identify the entity based on corresponding distances between the minutes and the plurality of entity clusters, wherein the minutes are provided to the text generation machine learning model.

3. The method of claim 1, wherein the generated image is displayed as a background image behind a video of one or more participants in the video communication session.

4. The method of claim 1, further comprising obtaining additional transcribed text from audio associated with the video communication session; outputting, using the text generation machine learning model, a further text prompt based on the additional transcribed text, wherein the further text prompt includes additional entities in the additional transcribed text; providing the further textual prompt to the image generation machine learning model; and The generated image is updated, using the image generation machine learning model, to be responsive to the further text prompt, wherein the updated generated image includes a depiction of the additional entity in the additional transcribed text.

5. The method of claim 1, wherein the video communications session is a live session and the method is performed multiple times during the live session using incremental audio received during periods between successive performances of the method.

6. The method of claim 5, wherein the entity comprises a plurality of entities, and the plurality of entities are transformed into other entities based on the incremental audio.

7. The method of claim 1, further comprising: generating a transcript of the transcribed text; as well as The transcript of the transcribed text is indexed with thumbnail versions of the generated images.

8. The method of claim 1, further comprising: scoring the entity set based on visual aspects associated with each entity in the entity set using the text generation machine learning model; Wherein outputting the text prompt comprises: outputting the text prompt having the entity associated with the highest score.

9. The method of claim 1 , wherein the entities are a plurality of entities, a first entity is based on audio associated with the video communication session from a first user, a second entity is based on audio associated with the video communication session from a second user, and the generated image depicts a logical connection between the first entity and the second entity.

10. The method of claim 1, further comprising: receiving, prior to said video communication session, a prewritten text; outputting one or more images based on entities detected in the prewritten text using the image generation machine learning model; detecting that the transcribed text matches a specific portion of the prewritten text; and A corresponding pre-generated image is caused to be displayed in the video communications session.

11. The method of claim 10, further comprising: Graphical data is generated for displaying a user interface, the user interface comprising a set of suggested backgrounds for use during the video communications session, wherein the set of suggested backgrounds includes the one or more images.

12. The method of claim 1, further comprising: An option is provided to save the generated image in association with the transcribed text of the video communication session.

13. The method of claim 1, further comprising: The transcribed text is deleted after the video communication session ends.

14. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: obtaining transcribed text from audio associated with the video communication session; providing the transcribed text to a text generation machine learning model; outputting a text prompt based on the transcribed text using the text generation machine learning model, wherein the text prompt includes entities in the transcribed text; providing the text prompt to an image generation machine learning model; generating, using the image generation machine learning model, a generated image responsive to the text prompt, wherein the generated image includes a depiction of the entity in the transcribed text; and The generated image is caused to be displayed in the video communications session.

15. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise identifying the entity in the transcribed text by: generating a transcript of the transcribed text; and The minutes are compared to a plurality of entity clusters to identify the entity based on corresponding distances between the minutes and the plurality of entity clusters, wherein the minutes are provided to the text generation machine learning model.

16. The non-transitory computer-readable medium of claim 14, wherein the generated image is displayed as a background image behind a video of one or more participants in the video communications session.

17. The non-transitory computer readable medium of claim 14, wherein the operations further comprise: obtaining additional transcribed text from audio associated with the video communication session; outputting, using the text generation machine learning model, a further text prompt based on the additional transcribed text, wherein the further text prompt includes additional entities in the additional transcribed text; providing the further textual prompt to the image generation machine learning model; and The generated image is updated, using the image generation machine learning model, to be responsive to the further text prompt, wherein the updated generated image includes a depiction of the additional entity in the additional transcribed text.

18. A computing device comprising: processor; as well as a memory coupled to the processor, the memory having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising: obtaining transcribed text from audio associated with the video communication session; providing the transcribed text to a text generation machine learning model; outputting a text prompt based on the transcribed text using the text generation machine learning model, wherein the text prompt includes entities in the transcribed text; providing the text prompt to an image generation machine learning model; generating, using the image generation machine learning model, a generated image responsive to the text prompt, wherein the generated image includes a depiction of the entity in the transcribed text; and The generated image is caused to be displayed in the video communications session.

19. The computing device of claim 18, wherein the operations further comprise identifying the entity in the transcribed text by: generating a transcript of the transcribed text; and The minutes are compared to a plurality of entity clusters to identify the entity based on corresponding distances between the minutes and the plurality of entity clusters, wherein the minutes are provided to the text generation machine learning model.

20. The computing device of claim 18, wherein the generated image is displayed as a background image behind a video of one or more participants in the video communications session.

Images