Generate images for video communication sessions

Abstract

The media application obtains transcribed text from audio associated with the video communication session. The media application provides the transcribed text to a text generation machine learning model. The text generation machine learning model outputs a text prompt based on the transcribed text, the text prompt including an entity in the transcribed text. The media application provides the text prompt to an image generation machine learning model. The image generation machine learning model outputs a generated image responsive to the text prompt, the generated image including a depiction of the entity in the transcribed text. The media application causes the generated image to be displayed in the video communication session.

Description

[Background technology]

[0001] Generating images for video conferences that occur in real time is difficult because topics can be diverse and can change rapidly. While users may retrieve images from a database and add them to a video conference, the time it takes to identify and retrieve the image may make the image inappropriate by the time a user finds and shares it.

[0002] Additionally, if a user saves a recording of a video communication session, the user may have difficulty remembering the subjects discussed in the video communication session, and this problem may be exacerbated with each additional video communication that the user saves.

[0003] The background art provided herein is described for purposes of generally presenting the context of the present disclosure. The work of the inventors named herein is not admitted expressly or impliedly as prior art to the present disclosure to the extent described in this background art section, including aspects of the description that may not qualify as prior art at the time of filing. Summary of the Invention

[0004] A computer-implemented method includes obtaining transcribed text from audio associated with a video communication session. The method further includes providing the transcribed text to a text generation machine learning model. The method further includes outputting a text prompt based on the transcribed text using the text generation machine learning model, the text prompt including an entity in the transcribed text. The method further includes providing the text prompt to an image generation machine learning model. The method further includes outputting a generated image responsive to the text prompt using the image generation machine learning model, the generated image including a depiction of the entity in the transcribed text. The method further includes displaying the generated image in the video communication session.

[0005] In some embodiments, the method further includes identifying entities from the transcribed text by generating a summary of the transcribed text and comparing the summary to a plurality of clusters of entities to identify the entities based on corresponding distances between the summary and the plurality of clusters of entities, wherein the summary is provided to the text-generating machine learning model. In some embodiments, the generated image is displayed as a background image behind videos of one or more participants in the video communication session.

[0006] In some embodiments, the method includes obtaining additional transcribed text from audio associated with the video communication session and outputting, using a text generation machine learning model, an additional text prompt based on the additional transcribed text. The additional text prompt includes an additional entity within the additional transcribed text. The method further includes providing the additional text prompt to an image generation machine learning model and updating, using the image generation machine learning model, a generated image responsive to the additional text prompt. The updated generated image includes a depiction of the additional entity within the additional transcribed text. In some embodiments, the video communication session is a live session, and the method is performed multiple times during the live session using incremental audio received during periods between successive executions of the method.

[0007] In some embodiments, the entity includes a plurality of entities, and the plurality of entities transition to other entities based on the incremental audio. In some embodiments, the method further includes generating a summary of the transcribed text and indexing the summary of the transcribed text with a thumbnail version of the generated image. In some embodiments, the method further includes scoring the set of entities based on a visual aspect associated with each entity of the set of entities using a text generation machine learning model, and outputting the text prompt includes outputting a text prompt including the entity associated with the highest score. In some embodiments, the entity is a plurality of entities, a first entity is based on audio from a first user associated with the video communication session, and a second entity is based on audio from a second user associated with the video communication session, and the generated image depicts a logical connection between the first entity and the second entity.

[0008] In some embodiments, the method further includes receiving pre-written text prior to the video communication session, using an image generation machine learning model to output one or more images based on entities detected in the pre-written text, detecting that the transcribed text matches a particular portion of the pre-written text, and displaying the corresponding pre-generated image in the video communication session. In some embodiments, the method further includes generating graphical data for displaying a user interface including a set of suggested backgrounds for use during the video communication session, the set of suggested backgrounds including one or more images. In some embodiments, the method further includes providing an option to save the generated image associated with the transcribed text of the video communication session. In some embodiments, the method further includes deleting the transcribed text after the video communication session has ended.

[0009] In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include obtaining transcribed text from audio associated with the video communication session, providing the transcribed text to a text generation machine learning model, and outputting, using the text generation machine learning model, a text prompt based on the transcribed text. The text prompt includes an entity within the transcribed text. The operations further include providing the text prompt to an image generation machine learning model and using the image generation machine learning model to generate a generated image responsive to the text prompt. The generated image includes a depiction of the entity within the transcribed text. The operations further include displaying the generated image in the video communication session.

[0010] In some embodiments, the operations further include identifying entities from the transcribed text by generating a summary of the transcribed text and comparing the summary to a plurality of clusters of entities to identify the entities based on corresponding distances between the summary and the plurality of clusters of entities, and providing the summary to the text-generating machine learning model. In some embodiments, the generated image is displayed as a background image behind videos of one or more participants in the video communication session. In some embodiments, the operations further include obtaining additional transcribed text from audio associated with the video communication session and outputting, using the text-generating machine learning model, a further text prompt based on the additional transcribed text. The further text prompt includes the additional entity in the additional transcribed text. The operations further include providing the further text prompt to the image-generating machine learning model and updating, using the image-generating machine learning model, the generated image responsive to the further text prompt. The updated generated image includes a depiction of the additional entity in the additional transcribed text.

[0011] In some embodiments, a computing device includes one or more processors and a memory coupled to the one or more processors. The memory has instructions stored therein. When executed by the processor, the instructions cause the processor to perform operations. The operations include obtaining transcribed text from audio associated with the video communication session, providing the transcribed text to a text generation machine learning model, and outputting, using the text generation machine learning model, a text prompt based on the transcribed text. The text prompt includes an entity within the transcribed text. The operations further include providing the text prompt to an image generation machine learning model and using the image generation machine learning model to generate a generated image responsive to the text prompt. The generated image includes a depiction of the entity within the transcribed text. The operations may further include displaying the generated image in the video communication session.

[0012] In some embodiments, the operations further include identifying entities from the transcribed text by generating a summary of the transcribed text and comparing the summary to a plurality of clusters of entities to identify the entities based on corresponding distances between the summary and the plurality of clusters of entities, wherein the summary is provided to the text-generating machine learning model. In some embodiments, the generated image is displayed as a background image behind videos of one or more participants in the video communication session. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of an example network environment for generating images for a video communication session, according to some embodiments described herein. [Figure 2] FIG. 1 is a block diagram of an example computing device for generating images for a video communication session, according to some embodiments described herein. [Figure 3]1 illustrates an exemplary user interface including a generated image according to some embodiments described herein. [Figure 4] 1 illustrates an exemplary user interface including a generated image of a background, according to some embodiments described herein. [Figure 5] 1 illustrates an exemplary user interface with pre-generated suggested backgrounds for use during a video communication session, according to some embodiments described herein. [Figure 6] 1 illustrates an exemplary user interface with thumbnail versions of generated images indexed according to a video communication session, according to some embodiments described herein. [Figure 7] 1 illustrates an example flowchart of a method for outputting a generated image for a video communication session according to some embodiments described herein. [Figure 8] 10 shows another example flowchart of a method for outputting a generated image for a video communication session according to some embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0014] Because video conference topics can be diverse and rapidly changing, generating images for video conferences occurring in real time can be challenging. The methods, systems, and non-transitory computer-readable media described herein generate images during a live video communication session, where the images represent the topics and conversations in the video communication session, update along with the audio during the session, and automatically provide relevant visual content. The described technology uses both text-generating and image-generating machine learning models to automatically generate relevant images, e.g., images depicting one or more entities discussed in the audio and / or text exchanged between participants in the video communication session. The described technology provides a technical effect by reducing the computational costs incurred when one or more participants in a video communication session perform an image search, preview multiple images, and select specific images to include in the video communication session. This technology is also advantageous because image content related to the subject of the live discussion is generated and displayed substantially in real time, which is not feasible with current manual image identification.

[0015] In some embodiments, the techniques may be implemented to generate images in advance of a video communication session (e.g., for storytelling, presentations, etc., where some aspects of the content of the video communication session are known in advance), and the generated images are automatically displayed in the video communication session based on matches between the session's live audio and / or text and previously generated images (and / or associated text). In this manner, the described techniques also save computational costs incurred during a video communication session by pre-caching relevant images, such that participants utilize little or no computational resources to perform searches, image previews, or image selections during the video communication session.

[0016] In some embodiments, the text-generating machine learning model transcribes text from a video communication session and outputs a text prompt that includes entities contained in the transcribed text. For example, a first user may discuss activities in Central Park last weekend, and a second user may discuss seeing an animated movie recently. The text-generating machine learning model may output a text prompt that includes the names of Central Park and the animated movie.

[0017] In some embodiments, the image-generating machine learning model receives a text prompt and outputs a generated image including a depiction of one or more entities in the transcribed text. For example, the image-generating machine learning model may output a generated image including a depiction of Central Park as it appears in an animated film. An initial image of Central Park may be generated based on the first user's audio and may be updated to modify the depiction of Central Park to a visual style that matches the visual style of the animated film. The generated image may also be used as a thumbnail image used to index a summary of the transcribed text for future searches. This may improve the ability to query data stores storing transcribed text (or other related content, such as audio / video recordings) and may increase efficiency by, for example, enabling the query process to operate via visual elements rather than dense text transcriptions. Users may easily recognize visual elements more quickly than text, and / or the search query process may be targeted to provide results according to the characteristics of the thumbnail image.

[0018] Exemplary Environment 100 FIG. 1 shows a block diagram of an exemplary environment 100 for generating images for a video communication session. In some embodiments, environment 100 includes a media server 101, user device 115a, and user device 115n coupled to a network 105. Users 125a, 125n may be associated with respective user devices 115a, 115n. In some embodiments, environment 100 may include other servers or devices not shown in FIG. 1. In FIG. 1 and the remaining figures, a letter following a reference number, e.g., "115a," indicates a reference to the element bearing that particular reference number. A reference number in text without a letter following it, e.g., "115," indicates a general reference to an embodiment of the element bearing that reference number.

[0019] The media server 101 may include a processor, memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to a network 105 via signal line 102. The signal line 102 may be a wired connection, such as Ethernet, coaxial cable, or fiber optic cable, or a wireless connection, such as Wi-Fi, Bluetooth, or other wireless technology. In some embodiments, the media server 101 transmits data to and receives data from one or more of the user devices 115a, 115n via the network 105. The media server 101 may include a media application 103a and a database 199.

[0020] The database 199 may store machine learning models, training data sets, video communication sessions (with user permission), generated images (with user permission), etc. The database 199 may also store social network data associated with the user 125, the user preferences of the user 125, etc.

[0021] The user device 115 may be a computing device that includes a memory coupled to a hardware processor. For example, the user device 115 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a mobile phone, a wearable device, a head-mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device that can access the network 105.

[0022] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored on user device 115a as media application 103b and / or on user device 115n as media application 103c. Signal lines 108 and 110 may be wired connections, such as Ethernet, coaxial cable, or fiber optic cable, or wireless connections, such as Wi-Fi, Bluetooth, or other wireless technologies. User devices 115a and 115n are accessed by users 125a and 125n, respectively. User devices 115a and 115n in FIG. 1 are used as an example. While FIG. 1 shows two user devices, 115a and 115n, this disclosure applies to system architectures having one or more user devices 115.

[0023] In some embodiments, the operations described herein are performed on the media server 101 and / or the user device 115. In some embodiments, some operations may be performed on the media server 101 and some may be performed on the user device 115.

[0024] The execution of actions is subject to user settings. For example, a user 125a may specify settings that actions are to be executed on each user device 115a and not on the media server 101. Such settings cause the actions described herein to be executed entirely on the user device 115a, and not on the media server 101. Furthermore, a user 125a may specify that the user's video and / or other data be stored only locally on the user device 115a, and not on the media server 101. Such settings result in user data not being transmitted to or stored on the media server 101. The transmission of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and the performance of actions on such data by the media server 101 are performed only if the user consents to the transmission, storage, and performance of actions by the media server 101. The user is provided with the option to change settings at any time, for example, to enable or disable the use of the media server 101.

[0025] Machine learning models (e.g., neural networks or other types of models) are stored locally on the user device 115 and utilized for one or more operations with specific user permission. Server-side models are used only with user permission. Additionally, trained models may be provided for use on the user device 115. During such use, on-device training of the model may be performed if permitted by the user 125. Updated model parameters may be sent to the media server 101 if permitted by the user 125, for example, to enable federated learning. The model parameters do not include any user data.

[0026] In some embodiments, the media application 103 may be implemented using hardware including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, the media application 103a may be implemented using a combination of hardware and software.

[0027] The media application 103 obtains transcribed text from audio associated with the video communication session. In some embodiments, the media application 103 includes a text-generating machine learning model that receives the transcribed text and outputs a text prompt based on the transcribed text. The text prompt includes an entity, such as a place, a person, or a video game, in the transcribed text.

[0028] The media application 103 includes an image generation machine learning model that receives a text prompt. The image generation machine learning model generates a generated image responsive to the text prompt. The generated image includes a depiction of an entity in the transcribed text.

[0029] Exemplary Computing Device 200 2 is a block diagram of an example computing device 200 that may be used to implement one or more features described herein. Computing device 200 may be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 200 is a media server 101 used to implement media application 103a. In another example, computing device 200 is a user device 115.

[0030] In some embodiments, computing device 200 includes a processor 235, memory 237, input / output (I / O) interface 239, microphone 241, speaker 243, display 245, camera 247, and storage device 249, all coupled via bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, microphone 241 may be coupled to bus 218 via signal line 228, speaker 243 may be coupled to bus 218 via signal line 230, display 245 may be coupled to bus 218 via signal line 232, camera 247 may be coupled to bus 218 via signal line 234, and storage device 249 may be coupled to bus 218 via signal line 236.

[0031] Processor 235 may be one or more processors and / or processing circuits that execute program code and control basic operations of computing device 200. A “processor” includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., a multiprocessor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuits for achieving a function, dedicated processors for performing neural network model-based processing, neural circuits, systems with processors optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more coprocessors that perform neural network processing. In some embodiments, processor 235 may be a processor that processes data to generate a probabilistic output; for example, the output generated by processor 235 may be inaccurate or accurate within a range from an expected output. Processing need not be limited to a particular geographic location or have time limitations. For example, a processor may perform its functions in real time, offline, in batch mode, etc. Portions of processing may be performed by different (or the same) processing systems at different times and in different locations. A computer may be any processor in communication with a memory.

[0032] Memory 237 is typically provided in computing device 200 for access by processor 235 and may be any suitable processor-readable storage medium suitable for storing instructions for execution by a processor or set of processors, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., and may be located separately from and / or integrated with processor 235. Memory 237 may store software operated on computing device 200 by processor 235, including media application 103.

[0033] Memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, a video library application, a video management application, a video gallery application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in several environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application having web pages, as a mobile application (“app”) running on a mobile computing device, etc.

[0034] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include videos used by a video library application, user actions identified by other applications 264 (e.g., social networking applications), etc.

[0035] I / O interface 239 may provide functionality that allows computing device 200 to interface with other systems and devices. The interfaced devices may be included as part of computing device 200 or may be separate and in communication with computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 249), and input / output devices may communicate through I / O interface 239. In some embodiments, I / O interface 239 may be connected to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, monitor, etc.).

[0036] The microphone 241 may include hardware for detecting sound. For example, the microphone 241 may detect ambient noise, people talking, music, etc. using a single microphone 241 that is part of the user device 115.

[0037] In some embodiments, microphone 241 includes additional hardware for processing audio captured while the user is recording video. An analog-to-digital converter may convert the analog electrical signal to a digital electrical signal. A digital signal processor may convert the digital electrical signal to a digital output signal, which is sent to speaker 243.

[0038] Speaker 243 may include hardware for generating audio signals heard by a user. In some embodiments, speaker 243 includes an amplifier used to amplify a particular channel, frequency, etc.

[0039] Display 245 includes hardware for displaying content of the output applications described herein, e.g., images, video, and / or user interfaces, and for receiving touch (or gesture) input from a user. For example, display 245 may be utilized to display a user interface including a set of suggested backgrounds for use during a video communication session. Display 245 may include any suitable display device, such as a liquid crystal display (LCD), light-emitting diode (LED), or plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touchscreen, a three-dimensional display screen, or other visual display device. For example, display 245 may be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or headset device, or a monitor screen of a computing device.

[0040] Camera 247 may be any type of image capture device capable of capturing images and / or video. In some embodiments, camera 247 captures images or video that I / O interface 239 sends to media application 103.

[0041] The storage device 249 stores data related to the media application 103. For example, the storage device 249 can store training datasets, text-generating machine learning models, image-generating machine learning models, videos (with user permission), summaries (with user permission), etc.

[0042] 2 illustrates an exemplary media application 103 that includes a video module 202, a text generation module 204, an image generation module 206, and an indexer 208. In some embodiments, each of the modules includes a set of instructions executable by a processor 235 to perform steps described in more detail below. In some embodiments, each of the components may be stored in memory 237 of the computing device 200 and accessible and executable by the processor 235.

[0043] Various embodiments described herein may include programmatic analysis of audio, video, text, or other media that is part of a video communication session. For example, audio may include spoken audio from participants in the video communication session (or other audio, such as recorded audio, audio detected by a participant's microphone, etc.), video may include video (e.g., from a camera) or other video content (e.g., a shared screen, streaming video, etc.) featuring participants, text may include chat messages exchanged between participants, and other media may include files (e.g., documents, images, multimedia objects, etc.). Programmatic analysis of audio, video, text, or other media (content) is performed with specific user permission from participants in the video communication session, such as a participant providing specific content, a participant receiving specific content, all participants in the video communication session, a moderator or host of the video communication session, etc. Participants are notified that such programmatic analysis may be performed and can choose to selectively enable or disable programmatic analysis. If a participant denies permission, programmatic analysis does not occur. Additionally, analyzed content is processed in accordance with applicable laws and regulations and in a secure manner (e.g., using encryption and / or other security technologies, locally on the user device, or centrally on a server). Content is not stored without the user's permission. Furthermore, the technology is disabled entirely for certain sets of users, such as users who do not meet age criteria, users associated with particular organizations whose organizational policies prevent programmatic analysis, etc.

[0044] The video module 202 facilitates video communication sessions. For example, the video module 202 may be stored on a server and may include instructions for receiving a first video stream from a first user device and transmitting the first video stream to a second user device. The video stream includes audio. In some embodiments, the video module 202 transcribes the audio into transcribed text. For example, the video module 202 may include a transcription machine learning model or other voice-to-text engine.

[0045] The video module 202 may obtain transcribed text from audio associated with the video communication session. The video module 202 may transmit the transcribed text to the text generation module 204.

[0046] In some embodiments, the video module 202 obtains additional transcribed text from audio associated with the video communication session. For example, the video communication session may be a live session, and the video module 202 may generate transcribed text along with incremental audio as additional audio is received during the video communication session. In some embodiments, the video module 202 generates the transcribed text repeatedly, such as every minute, every five minutes, after each person speaks a word or sentence. The video module 202 can send the additional transcribed text to the text generation module 204 as the transcribed text is generated.

[0047] In some embodiments, the text generation module 204 generates a summary from the transcribed text. The summary may include a list of participants, entities discussed in the transcribed text (e.g., Sarah went to the Natural History Museum next to Central Park on Sunday), emotions associated with the entities (e.g., Sarah had a great time), etc. In this example, the entities may include "Sarah," "Natural History Museum," "Central Park," and "Sunday," and emotions include "fun," "happy," etc. (as associated with the text "having a great time").

[0048] In some embodiments, the text generation module 204 includes a text generation machine learning model that receives the transcribed text as input and outputs a summary. The text generation machine learning model may be a large-scale language model (LLM). In some embodiments, the text generation module 204 compares the summary to multiple clusters of entities and identifies one or more entities in the summary based on corresponding distances between the summary and the multiple clusters of entities. In some embodiments, the text generation module 204 uses a knowledge graph containing information about the entities to supplement the summary. The knowledge graph may be part of the media application 103 or a third-party service. The summary may be provided to the text generation machine learning model instead of the transcribed text.

[0049] In some embodiments, the text generation module 204 uses a text-generating machine learning model to output a text prompt based on the transcribed text, or, if the text generation module 204 also includes a summary, based on the summary. The text prompt includes one or more entities from the transcribed text. Continuing with the example above, the text-generating machine learning model may receive a summary and / or transcribed text in which Sarah writes that she went to the museum on Sunday and had a great time. The text-generating machine learning model may output a text prompt requesting an image of an old museum building made of bricks, adjacent to an overgrown garden with ivy growing on the museum's bricks. The text prompt may request an old museum building based on the term "natural history museum." Conversely, if the museum is a modern art museum, the text prompt may include "20th or 21st century art style."

[0050] In some embodiments, the text-generating machine learning model is trained by the text generation module 204 and may include one or more model forms or structures. For example, the model forms or structures may include any type of neural network, such as a linear network, a deep learning neural network implementing multiple layers (e.g., a linear network with "hidden layers" between input and output layers), a convolutional neural network (e.g., a network that divides or partitions input data into multiple portions or tiles, processes each tile separately using one or more neural network layers, and aggregates the results of processing each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data as input, such as words in a sentence or frames of a video, and outputs a sequence of results), etc.

[0051] The model format or structure may specify the connectivity between various nodes and their organization into layers. For example, nodes in a first layer (e.g., input layer) may receive transcribed text as input data or application data 266. For example, if the trained model is to be used to analyze text, such data may include, for example, one or more words or phrases per node. Subsequent intermediate layers may receive as input the outputs of nodes in previous layers according to the connectivity specified in the model format or structure. These layers may also be referred to as hidden layers. The final layer (e.g., output layer) generates the output of the machine learning model, such as a text prompt. In some embodiments, the model format or structure also specifies the number and / or type of nodes in each layer.

[0052] In some embodiments, the text-generating machine learning model identifies a set of entities in the transcribed text and outputs a score associated with each entity. In some embodiments, the text-generating machine learning model outputs a higher score for more visual entities compared to less visual entities. For example, in "I saw sunflowers," a higher score may be associated with the entity "sunflowers" compared to the score associated with the entity "music" in "I heard good music." If both types of entities occur near each other, such as in "I heard good music at the cafe," the entity "cafe" may be associated with a higher score than the entity "music." In some embodiments, more recent entities discussed in the transcribed text are given higher priority than older entities discussed in the transcribed text. The text-generating machine learning model may rank the set of entities based on their corresponding scores and output a text prompt including the entity associated with the highest score. In some embodiments, scoring is performed by a middle layer of a neural network.

[0053] In some embodiments, the text generation module 204 may include multiple trained text-generating machine learning models. One or more of the text-generating machine learning models may include multiple nodes arranged in layers by model structure or format. In some embodiments, a node may be a memoryless computational node configured, for example, to process a single unit of input and generate a single unit of output. The computation performed by the node may include, for example, multiplying each of multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias value or intercept value to generate the node output. In some embodiments, the computation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such computation may include operations such as matrix multiplication. In some embodiments, computations by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, individual processing units of a graphics processing unit (GPU), or dedicated neural circuitry. In some embodiments, a node may include memory, for example, capable of storing and using one or more previous inputs when processing a subsequent input. For example, a node with memory may include a long short-term memory (LSTM) node, which may use memory to maintain a "state" that allows the node to operate like a finite state machine (FSM).

[0054] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, the model may begin as multiple nodes organized into layers, as specified by the model format or model structure. At initialization, respective weights may be applied to the connections between each pair of nodes connected according to the model format, e.g., nodes in successive layers of a neural network. For example, each weight may be randomly assigned or initialized to a default value. The text-generating machine learning model may then be trained, e.g., using training data, to generate results.

[0055] The training may be performed using supervised learning techniques. In supervised learning, the training data may include multiple inputs (e.g., multiple transcribed text documents) and corresponding ground truth outputs (e.g., text prompts for each transcribed text document) for each input. Based on a comparison of the model's output (e.g., predicted text prompts) and the ground truth outputs (e.g., ground truth summaries), the values ​​of the weights are automatically adjusted, for example, in a manner that increases the probability that the model will generate a ground truth channel.

[0056] In some embodiments, the training is unsupervised. The text may be divided into clusters, and the clusters may be organized according to text similarity.

[0057] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained text-generating machine learning model may include an initial set of weights, for example, downloaded from a server that provides the weights. In various embodiments, the trained text-generating machine learning model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the text generation module 204 can generate a trained text-generating machine learning model based on prior training, for example, by the developer of the text generation module 204, a third party, etc.

[0058] In some embodiments, if the text-generating machine learning model includes a convolutional neural network trained using supervised learning, training the text-generating machine learning model may include obtaining a text prompt based on the transcribed text for each training session. The text-generating machine learning model may calculate a loss value for the transcribed text based on a comparison of the predicted text prompt with a ground truth text prompt (included in the training data). The text-generating machine learning model may update weights of one or more nodes of the convolutional neural network based on the loss value (e.g., so that the loss value decreases until it is below a threshold after performing another cycle of training and training). In some embodiments, the text-generating machine learning model includes trainable convolutional encoder and decoder layers with a time-domain convolutional network masking network.

[0059] Once trained, the text generation machine learning model receives the transcribed text (from the video communication session) as input and outputs the text prompt. The text generation module 204 provides the text prompt as input to the image generation module 206.

[0060] In some embodiments, the text generation machine learning model receives additional transcribed text from the video module 202 and generates further text prompts based on the additional transcribed text.

[0061] The image generation module 206 may include an image generation machine learning model that receives the text prompt as input and outputs a generated image that responds to the text prompt and includes a depiction of an entity in the transcribed text.

[0062] In some embodiments, the image generation module 206 trains an image generation machine learning model using training data that includes text prompts as input to generate images as ground truth data.

[0063] The image-generating machine learning model may be an autoregressive text-to-image generative model that generates images that support context-rich synthesis, including complex composition and world knowledge. In some embodiments, the image-generating machine learning model encodes images as a sequence of discrete tokens.

[0064] Alternatively, the image generation machine learning model may output the generated image using a diffusion model. The diffusion model may perform text conditioning of the text prompt. For example, if the text request is to replace the jacket worn by the subject in the initial image with a blue shirt, the diffusion model performs text conditioning by generating a blue shirt.

[0065] A diffusion model can perform a diffusion process on a noisy image. The diffusion model is trained by adding noise to the image and training the diffusion model to remove the noise through a denoising process. In practice, the diffusion model applies the denoising process to a random seed to generate a realistic image. By simulating diffusion, the diffusion model generates a noisy image and then performs dediffusion. Dediffusion is the process of an output image emerging from the noise.

[0066] In some embodiments, the diffusion model performs forward diffusion, which combines first performing de-diffusion to create a noisy image, providing the noisy image to a convolutional neural network with a self-attention mechanism to perform feature extraction, and then text conditioning the noisy image to generate an output image that satisfies the text prompt provided as input to the diffusion model. In some embodiments, the diffusion model performs de-diffusion using a de-noising diffusion implicit model (DDIM) inversion.

[0067] The trained image-generating machine learning model receives text prompts from the text-generating machine learning model, and the image-generating machine learning model responds to the text prompts and outputs generated images that include depictions of entities in the transcribed text.

[0068] In some embodiments, the image-generating machine learning model receives additional text prompts, and the generated image using the image-generating machine learning model is updated in response to the additional text prompts. The image-generating machine learning model can gradually update the generated image over time while retaining the depiction of each entity in the initial generated image. For example, a first summary may include, "There is a star called Earth," and the generated image is of Earth. A second summary may include, "Dinosaurs lived on it," and the updated generated image includes Earth with dinosaurs on its surface. A third summary may include, "A meteor shower hit Earth," and the updated generated image includes Earth with dinosaurs that were hit by the meteor shower. A fourth summary may include, "The dinosaurs became extinct and Earth became a fireball," and the updated generated image includes a fireball.

[0069] Once the video communication session ends, the text generation machine learning model may generate a summary of the transcribed text. The summary may include a consolidation of previous summaries generated repeatedly during the live video communication session, or the summary may be generated based on the fully transcribed text representing the entire video communication session. In some embodiments, the indexer 208 may index the transcribed text summary using the generated image. For example, the transcribed text summary may be indexed with a thumbnail version of the generated image.

[0070] In some embodiments, if the generated image is updated over time, the image generation module 206 can generate a video clip of the generated image and update the generated image that the indexer 208 indexes as a thumbnail version of the video clip associated with the summary. Advantageously, the thumbnail version of the generated image or video clip allows a user to quickly identify a particular video communication session based on viewing the thumbnail version.

[0071] The indexer 208 saves summaries and corresponding thumbnail versions of generated images with a particular user's permission. In some embodiments, the summaries are discarded after the completion of the video communication session unless the user provides permission to save the summaries. In some embodiments, the indexer 208 provides the user with an option to save the generated image(s) in association with the transcribed text of the video communication session and indexes the generated images in response to receiving a selection of the option from the user. In some embodiments, if the user does not provide permission to save the summaries, the indexer 208 deletes the transcribed text and / or summaries after the video communication session ends. In some embodiments, if the user does not provide permission to save the generated images, the image generation module 206 may generate any generated images.

[0072] In addition to, or instead of, using the generated images in indexing the above summaries, they may also be used to index other content associated with a video communication session that may be stored, such as audio and / or visual recordings of the video communication session. Indeed, generating image indexing files in this manner may not be limited to video communication sessions and may also apply, for example, to audio communication sessions that lack any visual components. However, it may be useful for the generated images to be displayed to the user during the communication session to facilitate recognition for later retrieval of the indexed content.

[0073] Exemplary User Interface In some embodiments, the transcribed text describes multiple entities. For example, a first user may describe that on their last vacation, they went on their first hot air balloon ride. A second user may describe that on their last vacation, they went backcountry skiing in the mountains. In this example, the transcribed text includes the following entities: the hot air balloon and the mountain where people go backcountry skiing.

[0074] The text generation module 204 receives the transcribed text and outputs a text prompt requesting a generated image including a hot air balloon and a mountain where a person goes backcountry skiing. The image generation module 206 receives the text prompt and outputs a generated image based on the text prompt, including the entities in the text prompt. In some embodiments, the generated image depicts a logical connection between a first entity and a second entity. For example, instead of a generated image including a hot air balloon the same size as a mountain, the generated image includes a hot air balloon sized and positioned such that the hot air balloon is part of the same scene as the mountain.

[0075] In some embodiments, the generated image is displayed in the video communication session, for example, the generated image is displayed while the user listens to audio associated with the video communication session.

[0076] 3 shows an exemplary user interface 300 that includes a generated image 307. The user interface 300 includes a video screen 305 with a generated image and a video communication session icon, such as a telephone icon, that, when selected, terminates the video communication session. The generated image 307 includes a hot air balloon 315 and a mountain 320 where a person is going backcountry skiing. As a result, the generated image advantageously combines visual aspects associated with entities from each person participating in the video communication session.

[0077] As the video communication session continues, the video module 202 obtains additional transcribed text from the audio associated with the video communication session. The text generation machine learning model receives the additional transcribed text as input and outputs additional text prompts. The image generation machine learning model receives the additional text prompts and updates the generated image in response to the additional text prompts. For example, continuing with the above details, the first user may further describe that they took a wine tasting tour after traveling in a hot air balloon.

[0078] The image generation machine learning model may update the generated image to include entities related to wine tasting, such as the rolling hills of Napa Valley, including the vineyards where the wine tasting event will be held. The image generation module 206 may update the generated image to show one or more of the entities transitioning to the additional entities. For example, a mountain may transition to the rolling hills of Napa Valley.

[0079] In some embodiments, the generated image is displayed as a background image behind the video of one or more participants in the video communication session. The image generation module 206 can output the generated image with a negative space in which the user's image is placed. In some embodiments, the image generation module 206 adjusts the brightness, contrast, and hue of the user's video stream so that the user appears to be in the environment.

[0080] 4 shows an exemplary user interface 400 including a generated image of a background. In this example, a first user 410 describes that last weekend they went to see an animated movie that takes place primarily underwater. A second user 415 describes how they spent the weekend walking through Central Park. The image generation machine learning model outputs a generated image 405 that depicts a portion of Central Park and adds underwater elements to the generated image.

[0081] In some embodiments, the generated image is displayed separately from each user's video. For example, a video conferencing session may include a first image of a first user, a second image of a second user, and a third image that is a generated image.

[0082] In some embodiments, instead of a text prompt, the image-generating machine learning model receives pre-written text as input and outputs one or more images based on entities detected in the pre-written text. The one or more images may be used as a set of suggested backgrounds for use during the video communication session. For example, a parent may want to use the video communication session to read a story to their son, and the set of backgrounds may include typical characters in the story, such as a princess, a prince, a castle, a monster, a knight, etc. The image-generating machine learning model can update the generated images based on additional transcribed text, so that the suggested backgrounds serve as a convenient starting point for the story, with entities transitioning to other entities as the parent reads the story.

[0083] 5 shows an example user interface 500 with pre-generated suggested backgrounds for use during a video communication session. In this example, an image generation machine learning model outputs three images based on pre-written text 505, 510, and 515.

[0084] In some embodiments, the image-generating machine learning model can output an image based on user input. For example, a user selects box 520 in FIG. 5 to provide an entity that the image-generating machine learning model uses directly to output the generated image, or an entity that the text-generating machine learning model receives as input and uses to generate a text prompt that the image-generating machine learning model uses to output the generated image.

[0085] In some embodiments, prior to the video communication session, the text-generating machine learning model receives pre-written text and outputs a text prompt for the pre-written text. The image-generating machine learning model can receive a text prompt for the pre-written text and output one or more images based on entities detected in the text prompt. Alternatively, the image-generating machine learning model can receive the pre-written text directly and output one or more images based on entities detected in the pre-written text.

[0086] During a video communication session, the text generation machine learning model may detect that the transcribed text matches a particular portion of pre-written text, and the image generation machine learning model may cause a corresponding pre-generated image to be displayed.

[0087] 6 shows an example user interface 600 with thumbnail versions of generated images 605, 610, 615, 620 indexed according to the video communication session. In some embodiments, selecting one of the thumbnails displays a corresponding summary.

[0088] Exemplary Flowchart 7 shows an example flowchart of a method 700 for outputting a generated image for a video communication session. Method 700 may be performed by computing device 200 of FIG. 2. In some embodiments, method 700 is performed by user device 115, media server 101, or is performed partially on user device 115 and partially on media server 101 of FIG. 1.

[0089] 7 may begin at block 702. In block 702, it is determined whether permission to access user data is received from the user. If permission is not received, block 702 may be followed by block 704. In block 704, a notice refusing to provide the generated image is displayed. If permission is received, block 702 may be followed by block 706.

[0090] Transcribed text is obtained from audio associated with the video communication session at block 706. Block 706 may be followed by block 708.

[0091] A text generation machine learning model is provided with the transcribed text at block 708. Block 708 may be followed by block 710.

[0092] At block 710, the text-generating machine learning model outputs a text prompt based on the transcribed text, the text prompt including an entity in the transcribed text. In some embodiments, the text-generating machine learning model identifies entities from the transcribed text by generating a summary of the transcribed text, comparing the summary to a plurality of entities, and identifying the entity based on a corresponding distance between the summary and a cluster of the plurality of entities. In some embodiments, the transcribed text includes a plurality of entities, and the method further includes scoring the set of entities based on a visual aspect associated with each entity in the set of entities, and outputting the text prompt includes outputting a text prompt including the entity associated with the highest score. In some embodiments, the entities are a plurality of entities, a first entity is based on audio from a first user associated with the video communication session, and a second entity is based on audio from a second user associated with the video communication session, and the generated image depicts a logical connection between the first entity and the second entity. Block 710 may be followed by block 712.

[0093] The text prompt is provided to the image generation machine learning model at block 712. Block 712 may be followed by block 714.

[0094] At block 714, the image generation machine learning model outputs a generated image responsive to the text prompt, the generated image including a depiction of an entity in the transcribed text. Block 714 may be followed by block 716.

[0095] At block 716, the generated image is displayed in the video communication session. In some embodiments, the generated image is displayed as a background image behind the videos of one or more participants in the video communication session. In some embodiments, the method further includes providing an option to save the generated image in association with the transcribed text of the video communication session (or other content associated with the video communication session, such as an audio and / or video recording). In some embodiments, a summary of the transcribed text and / or other content associated with the video communication session is indexed with a thumbnail version of the generated image. In some embodiments, the transcribed text is deleted after the video communication session ends.

[0096] In some embodiments, the method further includes obtaining additional transcribed text from the audio associated with the video communication session. The text-generation machine learning model outputs a further text prompt based on the additional transcribed text, the further text prompt including the additional entity in the additional transcribed text. In response to the further text prompt, the further text prompt is provided to the image-generation machine learning model, and the updated generated image includes a depiction of the additional entity in the additional transcribed text.

[0097] In some embodiments, the video communication session is a live session and the method is performed multiple times during the live session with incremental audio received during periods between successive executions of the method. Further, instead of one entity, multiple entities are identified and one or more of the multiple entities transition to one or more of the multiple entities based on the incremental audio.

[0098] In some embodiments, prior to the video communication session, the method further includes receiving pre-composed text, the image generation machine learning model outputting one or more entities based on the entities detected in the pre-composed text, detecting that the transcribed text matches a particular portion of the pre-composed text, and displaying a corresponding pre-generated image. The method may further include generating graphics data for displaying a user interface including a set of suggested backgrounds for use during the video communication session.

[0099] 8 shows another example flowchart of a method 800 for outputting a generated image for a video communication session. Method 800 may be performed by computing device 200 of FIG. 2. In some embodiments, method 800 is performed by user device 115, media server 101, or performed partially on user device 115 and partially on media server 101 of FIG. 1.

[0100] 8 may begin at block 802. In block 802, it is determined whether permission to access user data is received from the user. If permission is not received, block 802 may be followed by block 804. In block 804, a notice refusing to provide the generated image is displayed. If permission is received, block 802 may be followed by block 806.

[0101] At block 806, text transcribed from audio associated with the video communication session is obtained. Block 806 may be followed by block 808.

[0102] The transcribed text is provided to a first layer of a text generation machine learning model at block 808. Block 808 may be followed by block 810.

[0103] At block 810, the text generation machine learning model outputs a summary based on the transcribed text, the summary including entities in the transcribed text. Block 810 may be followed by block 812.

[0104] In block 812, the summary is provided to a second layer of a text generation machine learning model. Block 812 may be followed by block 814.

[0105] The text generation machine learning model outputs a text prompt based on the summary at block 814. Block 814 may be followed by block 816.

[0106] The text prompt is provided to the image generation machine learning model at block 816. Block 816 may be followed by block 818.

[0107] At block 818, the image generation machine learning model outputs a generated image responsive to the text prompt, the generated image including a depiction of an entity in the transcribed text. Block 818 may be followed by block 820.

[0108] The generated image is displayed in the video communication session at block 820. Block 820 may be followed by block 822.

[0109] At block 822, the transcribed text and summary are deleted in response to the video communication session ending.

[0110] In addition to the above, users may be provided with controls that allow them to make choices about both whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about a user's activities, occupation, user preferences, or current location, such as using video communication session data, generating transcribed text, generating summaries, generating generated images, using generative artificial intelligence, storing data, etc.), as well as whether content or communications are sent to the user from a server. In the case of a video communication session, all participants in the video communication session consent to the use of such data. Furthermore, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, a user's identity may be processed such that personally identifiable information cannot be determined about the user, or if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be determined. Thus, users may control what information is collected about them, how that information is used, and what information is provided to them.

[0111] In the preceding description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the specification. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In some instances, structures and devices are shown in block diagram form in order to avoid obscuring the description. For example, embodiments may be described above primarily with reference to user interfaces and specific hardware. However, embodiments may apply to any type of computing device capable of receiving data and commands, and any peripheral device that provides services.

[0112] A reference herein to "some embodiments" or "some instances" means that a particular feature, structure, or characteristic described in connection with an embodiment or instance may be included in at least one implementation of the description. The appearances of the phrase "in some embodiments" in various places in the specification do not necessarily all refer to the same embodiments.

[0113] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. These steps are operations requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, principally for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0114] It should be recognized, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specified, as will be apparent from the discussion that follows, throughout this specification, discussions utilizing terms including "processing," "calculating," "computing," "determining," or "displaying," etc. will be understood to refer to operations and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in the computer system's registers or memory into other data similarly represented as physical quantities in the computer system's memory or registers, or other information storage, transmission, or display device.

[0115]

[0013] Embodiments herein may also relate to a processor for performing one or more steps of the above-described methods. The processor may be a special-purpose processor selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a non-transitory computer-readable storage medium, including, but not limited to, an optical disk, a ROM, a CD-ROM, a magnetic disk, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory including a USB key with non-volatile memory, or any type of disk, including any type of media suitable for storing electronic instructions, each connected to a computer system bus.

[0116] This specification may take the form of some entirely hardware embodiments, some entirely software embodiments, or some embodiments containing both hardware and software elements, hi some embodiments, this specification is implemented in software, including but not limited to firmware, resident software, microcode, etc.

[0117] Furthermore, the descriptions may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable medium may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, instruction execution apparatus, or instruction execution device.

[0118] A data processing system suitable for storing or executing program code includes at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory used during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code must be retrieved from bulk storage during execution.

Claims

1. 1. A computer-implemented method comprising: obtaining transcribed text from audio associated with the video communication session; providing the transcribed text to a text generation machine learning model; and using the text-generating machine learning model to output a text prompt based on the transcribed text, the text prompt including an entity in the transcribed text, the method comprising: providing the text prompt to an image generation machine learning model; and and further comprising using the image generation machine learning model to output a generated image responsive to the text prompt, the generated image including a depiction of the entity in the transcribed text, the method comprising: displaying the generated image in the video communication session; The computer-implemented method further comprising:

2. the entities from the transcribed text; generating a summary of the transcribed text; and comparing the summary to a plurality of clusters of entities and identifying the entities based on corresponding distances between the summary and the plurality of clusters of entities; 10. The method of claim 1, further comprising: identifying the summary by:

3. The method of claim 1 , wherein the generated image is displayed as a background image behind the video of one or more participants in the video communication session.

4. obtaining additional transcribed text from audio associated with the video communication session; and and outputting, using the text-generating machine learning model, a further text prompt based on the additional transcribed text, the further text prompt including an additional entity in the additional transcribed text, the method further comprising: providing the further text prompt to the image-generating machine learning model; and 10. The method of claim 1, further comprising: using the image generation machine learning model to update the generated image responsive to the additional text prompt, wherein the updated generated image includes a depiction of the additional entity in the additional transcribed text.

5. 10. The method of claim 1, wherein the video communication session is a live session, and the method is performed multiple times during the live session with incremental audio received during periods between successive executions of the method.

6. The method of claim 5 , wherein the entity comprises a plurality of entities, the plurality of entities transitioning to other entities based on the incremental audio.

7. generating a summary of the transcribed text; and indexing the summary of the transcribed text with a thumbnail version of the generated image; The method of claim 1 further comprising:

8. using the text-generating machine learning model to score the set of entities based on a visual aspect associated with each entity in the set of entities; The method of claim 1 , wherein outputting the text prompt comprises outputting the text prompt having the entity associated with the highest score.

9. the entity is a plurality of entities; the first entity is based on audio from a first user associated with the video communication session; the second entity is based on audio from a second user associated with the video communication session; The method of claim 1 , wherein the generated image depicts a logical connection between the first entity and the second entity.

10. receiving pre-written text prior to said video communication session; using the image-generating machine learning model to output one or more images based on entities detected in the pre-written text; Detecting that the transcribed text matches a particular portion of the pre-written text; and displaying a corresponding pre-generated image in said video communication session; The method of claim 1 further comprising:

11. generating graphical data for displaying a user interface including a set of suggested backgrounds for use during the video communication session; The method of claim 10 , wherein the set of suggested backgrounds includes the one or more images.

12. The method of claim 1 , further comprising: providing an option to save the generated image in association with the transcribed text of the video communication session.

13. The method of claim 1 , further comprising deleting the transcribed text after the video communication session ends.

14. 1. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: obtaining transcribed text from audio associated with the video communication session; providing the transcribed text to a text generation machine learning model; and using the text-generating machine learning model to output a text prompt based on the transcribed text, the text prompt including an entity in the transcribed text, and the operations include: providing the text prompt to an image generation machine learning model; and and further comprising: using the image generation machine learning model to generate a generated image responsive to the text prompt, the generated image including a depiction of the entity in the transcribed text, the operation comprising: displaying the generated image in the video communication session; 10. A non-transitory computer-readable medium, further comprising:

15. The operations include: extracting the entities from the transcribed text; generating a summary of the transcribed text; and comparing the summary to a plurality of clusters of entities and identifying the entities based on corresponding distances between the summary and the plurality of clusters of entities; 15. The non-transitory computer-readable medium of claim 14, further comprising identifying the summary by:

16. The non-transitory computer-readable medium of claim 14 , wherein the generated image is displayed as a background image behind videos of one or more participants in the video communication session.

17. The operation is obtaining additional transcribed text from audio associated with the video communication session; and and further comprising using the text-generating machine learning model to output a further text prompt based on the additional transcribed text, the further text prompt including an additional entity in the additional transcribed text, the operations comprising: providing the further text prompt to the image-generating machine learning model; and 15. The non-transitory computer-readable medium of claim 14, further comprising: using the image generation machine learning model to update the generated image responsive to the further text prompt, wherein the updated generated image includes a depiction of the additional entity in the additional transcribed text.

18. 1. A computing device comprising: a processor, and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform operations, the operations including: obtaining transcribed text from audio associated with the video communication session; providing the transcribed text to a text generation machine learning model; and using the text-generating machine learning model to output a text prompt based on the transcribed text, the text prompt including an entity in the transcribed text, and the operations include: providing the text prompt to an image generation machine learning model; and and further comprising: using the image generation machine learning model to generate a generated image responsive to the text prompt, the generated image including a depiction of the entity in the transcribed text, the operation comprising: displaying the generated image in the video communication session; The computing device further includes:

19. The operations include: extracting the entities from the transcribed text; generating a summary of the transcribed text; and comparing the summary to a plurality of clusters of entities and identifying the entities based on corresponding distances between the summary and the plurality of clusters of entities; 20. The computing device of claim 18, further comprising: identifying the summary by:

20. The computing device of claim 18 , wherein the generated image is displayed as a background image behind videos of one or more participants in the video communication session.

Images