Video context aware editing agent

Abstract

Techniques for using an artificial intelligent (AI) agent to modify audiovisual data is described herein. The system can obtain audiovisual data descriptive of one or more segments of audiovisual content. Additionally, the system can generate a semantic contextual description of each of the one or more segments of audiovisual content, the semantic contextual description generated based on one or more content understanding models. Moreover, the system can provide the semantic contextual description to a user. Furthermore, the system can receive a prompt from the user comprising one or more user-requested modifications to the semantic contextual description. The system can provide the prompt from the user and the semantic contextual description to the one or more content understanding models. Subsequently, the system can receive a second semantic contextual description from the one or more content understanding models, the second semantic contextual description reflecting the one or more user-requested modifications.

Description

VIDEO CONTEXT AWARE EDITING AGENTFIELD

[0001] The present disclosure relates generally to video context aware editing agents. More particularly, the present disclosure relates to systems and methods for editing audiovisual data using a content understanding agent.BACKGROUND

[0002] A computer can execute instructions to generate outputs provided some input(s) according to a parameterized model. The computer can use an evaluation metric to evaluate its performance in generating the output with the model. The computer can update the parameters of the model based on the evaluation metric to improve its performance. In this manner, the computer can iteratively “learn” to generate the desired outputs. The resulting model is often referred to as a machine-learned model.

[0003] A machine-learned model such as a large language model (LLM) can enable users to generate content via a natural language prompt. The model can be trained on massive amounts of text data to leam patterns and entity relationships in the language. The model can perform many types of language tasks and can understand textual data, identify entities and relationships between them, and generate new text.

[0004] A system having an artificial intelligent (Al) agent, also known as an intelligent agent, can perceive its environment, make decisions, and take actions to achieve specific goals. The system having an Al agent can be designed to mimic human-like cognitive abilities, such as reasoning, learning, problem-solving, and interacting with their surroundings.SUMMARY

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0006] One example aspect of the present disclosure is directed to a computer- implemented method for editing audiovisual data. The method can include obtaining, by a computing system, original audiovisual data descriptive of one or more segments of audiovisual content. Additionally, the method can include generating, using a transcript generation model, transcript data of each of the one or more segments of audiovisual content. Moreover, the method can include providing, on a user interface, the transcript data to a user.Furthermore, the method can include generating edited transcript data based on a prompt received from the user, the prompt comprising one or more user-requested modifications to the transcript data. Subsequently, the method can include processing, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

[0007] In some instances, the edited transcript data can be generated using the transcript generation model, wherein the transcript generation model comprises a disfluency detection model, and wherein disfluency labeled data is removed from the transcript data to generate the edited transcript data.

[0008] In some instances, the transcript generation model can be a speech-to-text model, a visual descriptor model, and / or an ambience descriptor model.

[0009] In some instances, the editing model can be a large language model (LLM).

[0010] In some instances, the transcript data can include text data descriptive of the audiovisual content of the audiovisual data. The text data can include disfluency labeled data, speech data, visual descriptor data, and / or ambience descriptor data.

[0011] In some instances, the one or more user-requested modifications can include a request to modify content of at least one of the one or more segments of audiovisual content, a request to remove at least one of the one or more segments of audiovisual content, and / or a request to shorten the audiovisual content. With the request to shorten the audiovisual content, the editing model can automatically remove at least one of the one or more segments of audiovisual content.

[0012] In some instances, the generation of the transcript data of each of the one or more segments of audiovisual content can include segmenting the audiovisual data into a plurality of audiovisual frames. Additionally, the method can include generating the transcript data for each of the plurality of audiovisual frames. Moreover, the method can include grouping the transcript data for each of the audiovisual frames into the transcript data for the one or more segments based on common content in the audiovisual frames.

[0013] In some instances, the prompt from the user can include a plain language prompt.

[0014] In some instances, the original audiovisual data and the modified audiovisual data can be presented to the user on a webpage.

[0015] One example aspect of the present disclosure is directed to a computing system having one or more processors and one or more non-transitory, computer-readable media storing instructions that, when implemented, cause the one or more processors to perform operations. The operations can include obtaining original audiovisual data descriptive of oneor more segments of audiovisual content. Additionally, the operations can include generating, using a transcript generation model, transcript data of each of the one or more segments of audiovisual content. Moreover, the operations can include providing, on a user interface, the transcript data to a user. Furthermore, the operations can include generating edited transcript data based on a prompt received from the user, the prompt comprising one or more user- requested modifications to the transcript data. Subsequently, the operations can include processing, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

[0016] Another example aspect of the present disclosure is directed to one or more non- transitory, computer-readable media storing instructions that, when implemented, cause one or more processors to perform operations. The operations can include obtaining original audiovisual data descriptive of one or more segments of audiovisual content. Additionally, the operations can include generating, using a transcript generation model, transcript data of each of the one or more segments of audiovisual content. Moreover, the operations can include providing, on a user interface, the transcript data to a user. Furthermore, the operations can include generating edited transcnpt data based on a prompt received from the user, the prompt comprising one or more user-requested modifications to the transcript data. Subsequently, the operations can include processing, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

[0017] Another example aspect of the present disclosure is directed to a computer- implemented method for editing audiovisual data. The method can include obtaining, by a computing system comprising one or more computing devices, audiovisual data descriptive of one or more segments of audiovisual content. Additionally, the method can include generating, by the computing system, a semantic contextual description of each of the one or more segments of audiovisual content, the semantic contextual description generated based on one or more content understanding models. Moreover, the method can include providing, by the computing system, the semantic contextual description to a user. Furthermore, the method can include receiving, by the computing system, a prompt from the user comprising one or more user-requested modifications to the semantic contextual description. The method can include providing, by the computing system, the prompt from the user and the semantic contextual description to the one or more content understanding models. Subsequently, the method can include receiving, by the computing system, a second semantic contextual description from the one or more content understanding models, the second semantic contextual description reflecting the one or more user-requested modifications.

[0018] Another example aspect of the present disclosure is directed to a computing system. The computing system can include one or more processors, and one or more non- transitory, computer-readable media storing instructions that, when implemented, cause the one or more processors to perform operations. The operations can include obtaining audiovisual data descriptive of one or more segments of audiovisual content. Additionally, the operations can include generating a semantic contextual description of each of the one or more segments of audiovisual content, the semantic contextual description generated based on one or more content understanding models. Moreover, the operations can include providing the semantic contextual description to a user. Furthermore, the operations can include receiving a prompt from the user comprising one or more user-requested modifications to the semantic contextual description. The operations can include providing the prompt from the user and the semantic contextual description to the one or more content understanding models. Subsequently, the operations can include receiving a second semantic contextual description from the one or more content understanding models, the second semantic contextual description reflecting the one or more user-requested modifications.

[0019] Another example aspect of the present disclosure is directed to one or more non- transitory, computer-readable media storing instructions that, when implemented, cause one or more processors to perform operations. The operations can include obtaining audiovisual data descriptive of one or more segments of audiovisual content. Additionally, the operations can include generating a semantic contextual description of each of the one or more segments of audiovisual content, the semantic contextual description generated based on one or more content understanding models. Moreover, the operations can include providing the semantic contextual description to a user. Furthermore, the operations can include receiving a prompt from the user comprising one or more user-requested modifications to the semantic contextual description. The operations can include providing the prompt from the user and the semantic contextual description to the one or more content understanding models. Subsequently, the operations can include receiving a second semantic contextual description from the one or more content understanding models, the second semantic contextual description reflecting the one or more user-requested modifications.

[0020] One example aspect of the present disclosure is directed to a computer- implemented method for editing audiovisual data. The method includes obtaining, by a computing system, original audiovisual data associated with audiovisual content. Additionally, the method includes processing the original audiovisual data, using an automatic speech recognition model, to generate raw transcript data of the audiovisual content. Furthermore, themethod includes processing the raw transcript, using a transcription generation model, to generate an edited transcript data. The edited transcript data optionally can have disfluency labeled data removed from the raw transcript. Subsequently, the method includes processing, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

[0021] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0022] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0024] Figure 1A depicts a block diagram of an example computing system that performs video editing operations according to example embodiments of the present disclosure.

[0025] Figure IB depicts a block diagram of an example computing device that performs video editing operations according to example embodiments of the present disclosure.

[0026] Figure 1 C depicts a block diagram of an example computing device that performs video editing operations according to example embodiments of the present disclosure.

[0027] Figure 2 depicts a diagram of an example user interaction with the system that performs audiovisual editing operations according to example embodiments of the present disclosure.

[0028] Figure 3 depicts an example system of a user interfacing with the Al agent to generate modified audiovisual data according to example embodiments of the present disclosure.

[0029] Figure 4 depicts a block diagram of an example system 400 for performing video editing operations according to example embodiments of the present disclosure.

[0030] Figure 5 depicts a flow chart diagram of an example method to perform video editing operations according to example embodiments of the present disclosure.

[0031] Figure 6 depicts a flow chart diagram of an example method to perform video editing operations according to example embodiments of the present disclosure.

[0032] Figure 7 depicts a flow chart diagram of an example method to perform video editing operations according to example embodiments of the present disclosure.

[0033] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations.DETAILED DESCRIPTIONOverview

[0034] Generally, the present disclosure is directed to a system having an artificial intelligent (Al) agent to perform video (e.g., audiovisual) editing operations. The Al agent can process a video to generate a transcript, receive user input to edit the transcript, and then makes or proposes edits to the video. For example, the transcript can include a representation of both the spoken content of the audiovisual material as well as textual descriptions of the non-speech and / or visual contents of the video.

[0035] The system can include an Al agent for performing media production task operations. The Al agent can utilize a large language model (LLM) to assist with performing the operations. The Al agent can perform video editing operations by incorporating artificial intelligence (Al) capabilities to assist in the process of video editing. The Al agents can be designed to streamline and enhance various aspects of video editing, making the process more efficient and accessible to users, whether they are professional editors or beginners.

[0036] The system can leverage the knowledge and intelligence embedded in foundation models, particularly LLMs, to allow the agent to be more flexible and proactive in the amount and kinds of help that the agent can offer users (e.g., content creators, consumers).

[0037] For example, the agent can perform the following operations: understanding user intent, requirements, and parameters expressed in natural language; create complex plans of action through LLM generative processes; communicate the plans, reasoning behind the plans, and setting expectations for the user about what the outcome of following the plans will be; and operating directly on media transcripts, altering them according to user specifications.

[0038] The system can enable advanced editing of media by non-experts. In some instances, the system can include a small, closed set of natural language processing (NLP) operations (e g., disfluency removal, grammar correction, and summarization) that are performed on transcripts of recorded audio followed by multimedia (e.g., audio, video) editing operations. The multimedia operations can be enhanced by the system using generative methods.

[0039] The system can perform operations, such as automated video editing, video enhancements, content analysis, speech recognition, graphics integration, sound editing, facial recognition, human emotion analysis, automated transitions, and time-saving features.

[0040] According to some embodiments, the system can analyze video footage and automatically create edited sequences based on predefined styles or templates. The system can identify key scenes, remove unwanted footage, and arrange clips to create a coherent video. The system can enhance video qualify by upscaling resolution, improving color grading, reducing noise, and stabilizing shaky footage. The system can analyze video content to identify objects, faces, or specific elements within the video. This analysis can be used for various purposes, such as automatic tagging, smart search, or scene recognition. The system can transcribe speech in videos, making it easier to edit and work with audio content. The system can also identify speakers, which can be helpful when dealing with interviews or multi-person dialogues. The system can intelligently integrate text, titles, subtitles, and graphics into videos, ensuring proper placement and alignment. The system can suggest appropriate music tracks or sound effects based on the video's mood or content, making it easier to find suitable audio elements. The system can detect and analyze facial expressions to help identify emotions in the video. This information can be used for emotion-based editing or creating personalized content. The system can recommend or apply transitions and visual effects that complement the style and content of the video. The system can automate repetitive tasks like video trimming, color correction, and filtering.

[0041] Additionally, the agent can include actuators (e g., generative model) for carrying out and augmenting the plans through the editing of input media. The agent can include tools that enable: performing classic operations (e.g., disfluency removal, summarization); command-line calls to media-editing binaries; generative model invocations; and API calls to services both to perform inference and to acquire information for reasoning and / or planning purposes. An example of a generative model invocation can be to utilize a large music model to add and / or modify background music to the audiovisual content.

[0042] Moreover, the agent can be multimodal and understand the visual / scene information in videos and be able to converse about it and potentially make plans based on the understanding. The agent can use generative models to directly make desired changes to audiovisual data (e.g., videos, audio).

[0043] In some instances, the agent can include text-only LLMs to understand language input and to perform complex operations. The operations can include; planning a high-level task having fine-grained steps for embodied robot task completion; driving a command-linebased on natural language descriptions of the desired outcomes; building pipeline specifications for combining previously unseen tool calls into fairly complex pipelines; query ing a tool or API to get grounded facts or desired information necessary for computing an output.

[0044] The system can improve user experience by providing fluent snippets or summaries which are derived from the full content. The content can be generated with disfluency removal. Additionally, the system can enable media clips to be returned instead of textual answers. The system can add the capability of cropping disfluent audio and add the capability to generate audio summaries. The system can receive edits to a transcript of video meeting recording, and the corresponding meeting video can be automatically updated. The system can automatically summarize incoming and / or outgoing voice messages. Content creators on video sharing platforms can edit the ASR transcript that is automatically generated for their video, and the system can automatically update their video to match the edits. This can save a user time from having to re-record their video.

[0045] In some embodiments, the system can be utilized to modify meeting recordings, podcasts, or vlogs where the speech is the primary focus of the content. For example, the system can be trained to optimize the coherence of the speech sounds and facial expressions and mouth movements so that the audio and video match.

[0046] In one example, a user can record a podcast where the user speaks extemporaneously on some recent occurrence. Because the delivery is not rehearsed, the user ends up correcting himself frequently and his speech is full of many mistakes. The system can process the audio with a disfluency removal filter, which scans the transcript for disfluencies and removes them, and then produces a recording of the speech where all the identified disfluencies have been removed smoothly. Subsequently, the user can realize that they have been using an incorrect word to describe the event, and the user changes all occurrences of the incorrect word to a new phrase. As a result, the audio is updated with the new phrase, and the video is updated based on the new phrase.

[0047] One of the advantages of this invention is that the system is designed to be multilingual, because a separate process edits the text, and the model can be trained to stitch together media independent of the language. The text can be edited manually by a user or automatically by a machine-learned model. The multilingual aspect of this invention significantly increases the potential impact of the invention.Example Devices and Systems

[0048] Figure 1A depicts a block diagram of an example computing system 100 that performs audiovisual editing operations according to example embodiments of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0049] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0050] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality' of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0051] In some implementations, the user computing device 102 can store or include one or more content understanding models 120. For example, the content understanding models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine- learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example content understanding models 120 are discussed with reference to Figures 2-6 can include a transcript generation model 121 and an audiovisual editing model 122.

[0052] In some implementations, the one or more content understanding models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implementmultiple parallel instances of a single model 120 (e.g., to perform parallel editing operations across multiple sections of a video).

[0053] More particularly, the transcript generation model 121 can include an automatic speech recognition (ASR) model, a scene recognition model, and so on. The audiovisual editing model 122 can be a video editing model that edits a video based on a modification to a transcript.

[0054] Additionally or alternatively, one or more content understanding models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the content understanding models 140 can be implemented by the server computing system 140 as a portion of a web service (e.g., a video editing service). Thus, one or more content understanding models 120 can be stored and implemented at the user computing device 102 and / or one or more content understanding models 140 can be stored and implemented at the server computing system 130.

[0055] Example content understanding models 140 are discussed with reference to Figures 2-6 can include a transcript generation model 141 and an audiovisual editing model 142. The transcript generation model 141 can include an ASR model, a scene recognition model, and so on. The audiovisual editing model 142 can be a video editing model that edits a video based on a modification to a transcript.

[0056] The user computing device 102 can also include one or more user input components 125 that receives user input. For example, the user input component 125 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0057] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0058] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0059] As described above, the server computing system 130 can store or otherwise include one or more content understanding models 140. For example, the content understanding models 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example content understanding models 140 are discussed with reference to Figures 2-6 can include a transcript generation model 141 and an audiovisual editing model 142. The transcript generation model 141 can include an ASR model, a scene recognition model, and so on. The audiovisual editing model 142 can be a video editing model that edits a video based on a modification to a transcript.

[0060] The user computing device 102 and / or the server computing system 130 can train the content understanding models 120 and / or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.

[0061] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0062] The training computing system 150 can include a model trainer 160 that trains the content understanding models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be back propagated through the model(s) to update one or more parameters of the model (s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0063] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0064] In particular, the model trainer 160 can train the content understanding models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, user input in response to original audiovisual data. The system can generate modified audiovisual data based on the user input. Additionally, the training data 162 can include additional user input in response to modified audiovisual data.

[0065] In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process can be referred to as personalizing the model.

[0066] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory. and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions that are stored in a tangible computer- readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0067] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 180 can be earned via any type of wired and / or wireless connection, using a widevariety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0068] The machine-learned models described in this specification may be used in a variety of tasks, applications, and / or use cases.

[0069] In some cases, the input includes audiovisual data, and the task is a computer vision task and / or a speech recognition task. Audiovisual data can include visual data and / or audio data.

[0070] In some cases, the input includes visual data, and the task is a computer vision task. The input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest. As another example, the image processing task can be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, a motion of the scene depicted at the pixel between the images in the network input.

[0071] In some cases, the input includes audio data representing a spoken utterance and the task is a speech recognition task. The output may comprise a text output which is mapped to the spoken utterance. In some cases, the task comprises encrypting or decrypting input data. In some cases, the task comprises a microprocessor performance task, such as branch prediction or memory address translation.

[0072] In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the imagedata, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine- learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.

[0073] In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine- learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.

[0074] In some implementations, the input to the machine-learned model(s) of the present disclosure can be speech data. The machine-learned model(s) can process the speech data to generate an output. As an example, the machine-learned model(s) can process the speech data to generate a speech recognition output. As another example, the machine-learned model(s) can process the speech data to generate a speech translation output. As another example, the machine-learned model(s) can process the speech data to generate a latent embedding output. As another example, the machine-learned model(s) can process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representationof the speech data, etc.). As another example, the machine-learned model(s) can process the speech data to generate an upscaled speech output (e.g., speech data that is higher quality than the input speech data, etc.). As another example, the machine-learned model(s) can process the speech data to generate atextual representation output (e.g., atextual representation of the input speech data, etc.). As another example, the machine-learned model(s) can process the speech data to generate a prediction output.

[0075] In some implementations, the input to the machine-learned model(s) of the present disclosure can be latent encoding data (e g., a latent space representation of an input, etc.). The machine-learned model(s) can process the latent encoding data to generate an output. As an example, the machine-learned model(s) can process the latent encoding data to generate a recognition output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reconstruction output. As another example, the machine-learned model(s) can process the latent encoding data to generate a search output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reclustering output. As another example, the machine-learned model(s) can process the latent encoding data to generate a prediction output.

[0076] In some implementations, the input to the machine-learned model(s) of the present disclosure can be statistical data. Statistical data can be, represent, or otherwise include data computed and / or calculated from some other data source. The machine-learned model(s) can process the statistical data to generate an output. As an example, the machine-learned model(s) can process the statistical data to generate a recognition output. As another example, the machine-learned model(s) can process the statistical data to generate a prediction output. As another example, the machine-learned model(s) can process the statistical data to generate a classification output. As another example, the machine-learned model(s) can process the statistical data to generate a segmentation output. As another example, the machine-learned model(s) can process the statistical data to generate a visualization output. As another example, the machine-learned model(s) can process the statistical data to generate a diagnostic output.

[0077] In some implementations, the input to the machine-learned model(s) of the present disclosure can be sensor data. The machine-learned model(s) can process the sensor data to generate an output. As an example, the machine-learned model(s) can process the sensor data to generate a recognition output. As another example, the machine-learned model(s) can process the sensor data to generate a prediction output. As another example, the machine- learned model(s) can process the sensor data to generate a classification output. As another example, the machine-learned model (s) can process the sensor data to generate a segmentationoutput. As another example, the machine-learned model(s) can process the sensor data to generate a visualization output. As another example, the machine-learned model(s) can process the sensor data to generate a diagnostic output. As another example, the machine-learned model(s) can process the sensor data to generate a detection output.

[0078] In some cases, the machine-learned model(s) can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data and the output may comprise compressed audio data. In another example, the input includes visual data (e.g. one or more images or videos), the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding for input data (e.g. input audio or visual data).

[0079] Figure 1A illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training dataset 162. In such implementations, the content understanding models 120 can be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 can implement the model trainer 160 to personalize the content understanding models 120 based on user-specific data.

[0080] Figure IB depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0081] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0082] As illustrated in Figure IB, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0083] Figure 1C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0084] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0085] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 1C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0086] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As illustrated in Figure 1C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0087] Figure 2 depicts a diagram of an example user interaction 200 with the system that performs audiovisual editing operations according to example embodiments of the present disclosure.

[0088] In this example, a user 210 can interface with an Al agent 230 by selecting an input recording 220 (e.g., audiovisual data) to edit. The Al agent 230 can include content understanding models 120, 140. The user 210 can request, via a user interface 225, the Al agent 230 to edit the input recording 220. The Al agent 230 can receive the input recording 220 and make a recommendation to the user 210. In this example, the Al agent 230 suggests that the length of the input recording 220 is too long, in which the user 210 responds by requesting the Al agent 230 to shorten the video. The Al agent 230 can generate an output recording (e.g., output video) based on the requested edits of the user 210. For example, based on the userrequest received via the user interface 225, the Al agent can generate an output recording 240. The output recording 240 can be based on the transcript and text descriptions 245 of the input recording. The Al agent 230 can perform video understanding by generating the transcript and text descriptions 245. The transcript and text descriptions 245 can be generated by tokenizing the input recording 220 to generate textual descriptions of the video content and interleave it with the speech transcripts.

[0089] FIG. 3 depicts an example system 300 of a user interfacing with the Al agent to generate modified audiovisual data according to example embodiments of the present disclosure.

[0090] As illustrated in FIG. 3, the Al agent 230 can include a conversational video editing tool having an editing model 320 that can act as both the user interface 310 (e.g., chat interface) and the controlling logic for the system. Examples of the editing model 320 can include the machine-learned model (s) 120, 140 as described in FIG. 1A. The editing model 320 can include an LLM.

[0091] The system 300 interactions can include the user 210 communicating with the Al agent 230 having the editing model 320. The user 210 and the Al agent 230 can have a shared editing model 320 for the desired edits. The desired edits can be performed by an editing plan 330. The editing plan 330 can be generated based on edits to the transcript data 360 of the original audiovisual data 340. The original audiovisual data 340 can be processed by a transcript generation model 350 (e.g., ASR model, scene understanding model) to generate the transcript data 360 The transcript data 360 can include scene-to-text data that is generated by the scene understanding model. The transcript data 360 can include a representation of both the spoken content of the audiovisual material as well as textual descriptions of the non-speech and / or visual contents of the video. The transcript data 360 can include semantic contextual description of the original audiovisual data 340.

[0092] In one embodiment, the user 210 can request, via the user interface 310, the Al agent 230 to edit (e.g., simplify, change words, summarize, remove disfluencies) the transcript data 360. In another embodiment, the user 210 can edit the transcript data 360 directly via the user interface 310. In yet another embodiment, the user 210 can request the Al agent 230 to directly edit the original audiovisual data 340 (e g., original video) using a prompt.

[0093] In some instances, when the editing model 320 (e.g., LLM) performs an edit to the original audiovisual data 340, the Al agent 230 can issue commands via an editing plan 330 which is applied to the original audiovisual data 340 through generative editing libraries 370 and editing tools 380 to generate modified audiovisual data 390. The generative editinglibraries 370 can be custom libraries. The editing tools 380 can be second or third-party editing tools. The modified audiovisual data 390 can be further refined as desired as illustrated by the dotted line in FIG. 3 from the original audiovisual data 340 to the modified audiovisual data 390.

[0094] In some instances, the editing model 320 can be trained and / or tuned using prompting techniques 315. For example, the system 300 can train the editing model 320 (e.g., LLM) to produce the correct output to the user 210 and / or the editing plan 330. The system 300 can train the editing model 320 by using prompting techniques 315, which can include prompt engineering techniques, fine-tuning techniques, grounding techniques, and / or memory techniques.

[0095] In some instances, the generative editing libraries 370 can ensure that modifications (e.g., cuts, splices) to the original audiovisual data 340 are clean. Additionally, the generative editing libraries 370 can add smart capabilities associated with noise reduction, background smoothing, music generation, and / or frame interpolation. Given that ASR timings may not be accurate, the Al agent 230 can align edited transcripts back to ASR transcripts to find timing data to index back into the media for the purposes of creating cuts and splices at the appropriate moments. The Al agent 230 can utilize the generative editing libraries to perform audio stitching and generation operations.

[0096] In one embodiment, the Al agent 230 can clean up a train of thought audio recording to make a podcast (e.g. removing disfluencies, removing uninteresting sections, cutting to a pre-specified length). In another embodiment, the Al agent 230 can create a short or teaser video from a larger video given instructions from the user (e.g., pull out everything about a particular topic, or pull out a few sections based on a natural language description). The Al agent 230 can ensure consistent high quality at stitching seams by using stitching techniques (e.g., generative model invocation).

[0097] The Al agent 230 can process videos by feeding summaries of the videos to the editing model 320. Additionally, the editing model 320 can query a plurality of sections of the transcript data 360 (e.g., transcript) to process each section separately.

[0098] The system 300 can produce higher quality versions of audio and / or video input by propagating text modifications to the source media. Given the raw audio or video and some signals for words or phrases in the transcript that should be removed, re-arranged, or changed, the system 300 can smoothly crop, stitch, and generate audio so that it sounds as though the edited transcript was what was originally said. For example, the system 300 can perform a summarization operation by extracting relevant audio clips corresponding to the summary andsmoothly concatenating them. The summarization operation can be extended to videos, where adjacent video segments should be temporally smoothed. For example, when a person is interviewed on a TV program, the transitions are smooth when certain segments of the raw video are cropped. The smoothing process could take into account speech prosody changes (e.g., high pitch, low pitch, loud, soft, loudness, fast, slow, duration) and can involve generative aspects.

[0099] With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail.Example Model Arrangements

[0100] Figure 4 depicts a block diagram of an example system 400 for performing video editing operations according to example embodiments of the present disclosure.

[0101] In some implementations, system 400 can include transcript generation model(s) 420. The transcript generation model(s) 420 can process the original audiovisual data 340 to generate transcript data 360. The system can present the transcript data 360 to a user via a user interface. The system can receive, via the user interface, a user prompt 430 to modify the transcript data 360. The system can modify the transcript data 360 based on the user prompt 430 to generate an edited transcript data 440.

[0102] In some other implementations, the transcript generation model(s) 420 can generate the transcript data 360, and then automatically (e g., without a user prompt 430) modify the transcript data 360 to generate an edited transcript data 440. In some instances, the system 400 can utilize services and tools for disfluency removal. For example, stuttering can be an interruption in the smooth flow or fluency of speech. Breaks or disruptions that occur in the flow of speech are labeled disfluencies by the transcript generation model(s) 420. Disfluency removal can be performed by training a transcript generation model 420 (e.g., a disfluency detection model) on disfluency labeled data and applying it as a separate component following another transcript generation model 420 (e.g., an ASR model) and prior to being inputted into the editing model(s) 410. The system 400 can include an initial application of the enhancement work by cleanly removing disfluencies from audio and video. In one embodiment, the system 400 can remove disfluencies through a video cropping mechanism. In another embodiment, the system 400 can enable other NLP-based transformations such as summarization. For example, a user could remove words or phrases from different portions ofthe ASR transcript, and the audio and video can automatically be updated, by the system, to reflect the changes.

[0103] Additionally, the system 400 can include an editing model(s) 410. The editing model(s) 410 can be trained to process original audiovisual data 340 and the edited transcript data 440 to generate the modified audiovisual data 390. In some instances, the editing model(s) 410 can include an audio model to process the audio data and a video model to process the video data. The editing model(s) 410 can include an LLM.

[0104] The system 400 can generate modified audiovisual data 390 based on the output of the editing model(s) 410. In some instances, the system 400 can present the modified audiovisual data 390 on a webpage (e.g., html page) or mobile application to enable a user to compare the original audiovisual data 340 (e.g., original video) with modified audiovisual data 390 (e.g., modified video). For example, the webpage can present comparisons of the original audiovisual data 340 in a first column, the audiovisual data with disfluencies removed in the second column, the audiovisual data with grammar corrected in the third column, and the modified audiovisual data 390 in the last column.Example Methods

[0105] Figure 5 depicts a flow chart diagram of an example method to perform video editing operations according to example embodiments of the present disclosure. Although Figure 5 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of method 500 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the present disclosure.

[0106] At 502, a computing system can obtain original audiovisual data descriptive of one or more segments of audiovisual content.

[0107] At 504, the computing system can generate, using a transcript generation model, transcript data of each of the one or more segments of audiovisual content. In some instances, the transcript generation model can include a speech-to-text model. In some instances, the transcript generation model can include a visual descriptor model. In some instances, the transcript generation model can include an ambience descriptor model.

[0108] In some instances, the generation of the transcript data of each of the one or more segments of audiovisual content includes the system segmenting the audiovisual data into a plurality of audiovisual frames. Additionally, the system can generate the transcript data foreach of the plurality of audiovisual frames. Moreover, the system can group the transcript data for each of the audiovisual frames into the transcript data for the one or more segments based on common content in the audiovisual frames.

[0109] In some instances, the transcript data can include text data descriptive of the audiovisual content of the audiovisual data, wherein the text data includes speech data, visual descriptor data, and ambience descriptor data.

[0110] At 506, the computing system can provide, on a user interface, the transcript data to a user.

[0111] At 508, the computing system can generate edited transcript data based on a prompt received from the user, the prompt comprising one or more user-requested modifications to the transcript data.

[0112] In some instances, the prompt from the user comprises a plain language prompt.

[0113] In some instances, the one or more user-requested modifications includes a request to modify content of at least one of the one or more segments of audiovisual content.

[0114] In some instances, the one or more user-requested modifications includes a request to remove at least one of the one or more segments of audiovisual content.

[0115] At 510, the computing system can process, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data. In some instances, the editing model can be a large language model (LLM).

[0116] In some instances, the prompt from the user comprises a plain language prompt.

[0117] Figure 6 depicts a flow chart diagram of an example method to perform video editing operations according to example embodiments of the present disclosure. Although Figure 6 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of method 600 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the present disclosure.

[0118] At 602, a computing system can obtain audiovisual data descriptive of one or more segments of audiovisual content.

[0119] At 604, the computing system can generate a semantic contextual description of each of the one or more segments of audiovisual content. The semantic contextual description can be generated based on one or more content understanding models. For example, the semantic contextual description can include text data descriptive of the audiovisual content of the audiovisual data.

[0120] In some instances, the one or more content understanding models can include a large language model (LLM), a speech-to-text model, a visual descriptor model, and / or an ambience descriptor model.

[0121] In some instances, the generation of the semantic contextual description of each of the one or more segments of audiovisual content can include the computing system segmenting the audiovisual data into a plurality of audiovisual frames. Additionally, the computing system can generate the semantic contextual description for each of the plurality of audiovisual frames. Moreover, the computing system can group semantic contextual description for each of the audiovisual frames into the semantic contextual description for the one or more segments based on common content in the audiovisual frames.

[0122] At 606, the computing system can provide the semantic contextual description to a user. For example, the semantic contextual description can be presented, by the computing system, on a display of a user device.

[0123] At 608, the computing system can receive a prompt from the user comprising one or more user-requested modifications to the semantic contextual description. For example, the prompt from the user can include a plain language prompt.

[0124] In some instances, the one or more user-requested modifications can include a request to modify content of at least one of the one or more segments of audiovisual content.

[0125] In some instances, the one or more user-requested modifications can include a request to remove at least one of the one or more segments of audiovisual content.

[0126] At 610 the computing system can provide the prompt from the user and the semantic contextual description to the one or more content understanding models.

[0127] At 614, the computing system can receive a second semantic contextual description from the one or more content understanding models, the second semantic contextual description reflecting the one or more user-requested modifications.

[0128] In some instances, the computing system can generate a second audiovisual data based on the audiovisual data and the second semantic contextual description.

[0129] Figure 7 depicts a flow chart diagram of an example method to perform video editing operations according to example embodiments of the present disclosure. Although Figure 7 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of method 600 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the present disclosure.

[0130] At 702, a computing system can obtain original audiovisual data associated with audiovisual content.

[0131] At 704, the computing system can process the original audiovisual data, using an automatic speech recognition model, to generate raw transcript data of the audiovisual content.

[0132] At 706, the computing system can process the raw transcript, using a transcription generation model, to generate an edited transcript data, the edited transcript data having disfluency labeled data removed from the raw transcript.

[0133] In some instances, the edited transcript data comprises text data descriptive of the audiovisual content of the audiovisual data, wherein the text data includes disfluency labeled data, speech data, visual descriptor data, and ambience descriptor data.

[0134] In some instances, the transcript generation model comprises a speech-to-text model.

[0135] In some instances, the transcript generation model comprises a visual descriptor model.

[0136] In some instances, the transcript generation model comprises an ambience descriptor model.

[0137] In some instances, the edited transcript data can be generated using the transcript generation model. The transcript generation model can include a disfluency detection model, and the disfluency labeled data can be removed from the raw transcript data to generate the edited transcript data.

[0138] At 708, the computing system can process, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

[0139] In some instances, the editing model comprises a large language model (LLM).

[0140] In some instances, the edited transcript data comprises text data descriptive of the audiovisual content of the audiovisual data, wherein the text data includes disfluency labeled data, speech data, visual descriptor data, and ambience descriptor data.

[0141] In some instances, method 700 further includes the computing system receiving a prompt from the user having a user-requested modification to the edited transcript data. For example, the user-requested modification can be a request to modify content of the audiovisual content or a request to remove content of the audiovisual content. The modified audiovisual data can be generated by processing, using the editing model, the edited transcript data, the user-requested modification, and / or the original audiovisual data.

[0142] In some instances, the user-requested modification includes shortening the audiovisual content, and the editing model automatically removes a segment of audiovisual content.

[0143] In some instances, the prompt from the user comprises a plain language prompt.

[0144] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken, and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0145] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure covers such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for editing audiovisual data, the method comprising: obtaining, by a computing system, original audiovisual data descriptive of one or more segments of audiovisual content; generating, using a transcript generation model, transcript data of each of the one or more segments of audiovisual content; providing, on a user interface, the transcript data to a user; generating edited transcript data based on a prompt received from the user, the prompt comprising one or more user-requested modifications to the transcript data; and processing, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

2. The computer-implemented method of claim 1, wherein the edited transcript data is generated using the transcript generation model, wherein the transcnpt generation model comprises a disfluency detection model, and wherein disfluency labeled data is removed from the transcript data to generate the edited transcript data.

3. The computer-implemented method of claim 1, wherein the transcript generation model comprises a speech-to-text model4. The computer-implemented method of claim 1, wherein the transcript generation model comprises a visual descriptor model.

5. The computer-implemented method of claim 1, wherein the transcript generation model comprises an ambience descriptor model.

6. The computer-implemented method of claim 1, wherein the editing model comprises a large language model (LTM).

7. The computer-implemented method of claim 1, wherein the transcript data comprises text data descriptive of the audiovisual content of the audiovisual data, wherein the text data includes disfluency labeled data, speech data, visual descriptor data, and ambience descriptor data.

8. The computer-implemented method of claim 1, wherein the one or more user- requested modifications comprise a request to modify content of at least one of the one or more segments of audiovisual content.

9. The computer-implemented method of claim 1, wherein the one or more user- requested modifications comprise a request to remove at least one of the one or more segments of audiovisual content.

10. The computer-implemented method of claim 1, wherein the one or more user- requested modifications comprise a request to shorten the audiovisual content, and wherein the editing model automatically removes at least one of the one or more segments of audiovisual content.

11. The computer-implemented method of claim 1, wherein generating, by the computing system, the transcript data of each of the one or more segments of audiovisual content comprises: segmenting the audiovisual data into a plurality of audiovisual frames; generating the transcript data for each of the plurality of audiovisual frames; and grouping the transcript data for each of the audiovisual frames into the transcript data for the one or more segments based on common content in the audiovisual frames.

12. The computer-implemented method of claim 1, wherein the prompt from the user comprises a plain language prompt.

13. The computer-implemented method of claim 1, wherein the original audiovisual data and the modified audiovisual data are presented to the user on a webpage.

14. A computing system, comprising: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising:obtaining original audiovisual data descriptive of one or more segments of audiovisual content; generating, using a transcript generation model, transcript data of each of the one or more segments of audiovisual content; providing, on a user interface, the transcript data to a user; generating edited transcript data based on a prompt received from the user, the prompt comprising one or more user-requested modifications to the transcript data; and processing, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

15. The computing system of claim 14, wherein the edited transcript data is generated using the transcript generation model, wherein the transcript generation model comprises a disfluency detection model, and wherein disfluency labeled data is removed from the transcript data to generate the edited transcript data.

16. The computing system of claim 14, wherein the transcript generation model comprises a speech-to-text model, a visual descriptor model, or an ambience descriptor model.

17. The computing system of claim 14, wherein the transcript data comprises text data descriptive of the audiovisual content of the audiovisual data, wherein the text data includes disfluency labeled data, speech data, visual descriptor data, and ambience descriptor data.

18. The computing system of claim 14, wherein the one or more user-requested modifications comprise a first request to modify content of at least one of the one or more segments of audiovisual content, or a second request to remove at least one of the one or more segments of audiovisual content.

19. The computing system of claim 14, wherein the one or more user-requested modifications comprise a request to shorten the audiovisual content, and wherein the editing model automatically removes at least one of the one or more segments of audiovisual content.

20. One or more non-transitory, computer-readable media storing instructions that, when implemented, cause one or more processors to perform operations, the operations comprising: obtaining original audiovisual data descriptive of one or more segments of audiovisual content; generating, using a transcript generation model, transcript data of each of the one or more segments of audiovisual content; providing, on a user interface, the transcript data to a user; generating edited transcript data based on a prompt received from the user, the prompt comprising one or more user-requested modifications to the transcript data; and processing, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

21. A computer-implemented method for editing audiovisual data, the method comprising: obtaining, by a computing system comprising one or more computing devices, audiovisual data descriptive of one or more segments of audiovisual content; generating, by the computing system, a semantic contextual description of each of the one or more segments of audiovisual content, the semantic contextual description generated based on one or more content understanding models; providing, by the computing system, the semantic contextual description to a user; receiving, by the computing system, a prompt from the user comprising one or more user-requested modifications to the semantic contextual description; providing, by the computing system, the prompt from the user and the semantic contextual description to the one or more content understanding models; and receiving, by the computing system, a second semantic contextual description from the one or more content understanding models, the second semantic contextual description reflecting the one or more user-requested modifications.

22. The computer-implemented method of claim 21, wherein the one or more content understanding models comprise a large language model (LLM).

23. The computer-implemented method of claim 21, wherein the semantic contextual description comprises text data descriptive of the audiovisual content of the audiovisual data.

24. The computer-implemented method of claim 21, wherein the one or more content understanding models comprise a speech-to-text model.

25. The computer-implemented method of claim 21, wherein the one or more content understanding models comprise a visual descriptor model.

26. The computer-implemented method of claim 21, wherein the one or more content understanding models comprise an ambience descriptor model.

27. The computer-implemented method of claim 21, further comprising generating, by the computing system, second audiovisual data based on the audiovisual data and the second semantic contextual description.

28. The computer-implemented method of claim 21, wherein the one or more user-requested modifications comprise a request to modify content of at least one of the one or more segments of audiovisual content.

29. The computer-implemented method of claim 21, wherein the one or more user-requested modifications comprise a request to remove at least one of the one or more segments of audiovisual content.

30. The computer-implemented method of claim 21, wherein generating, by the computing system, the semantic contextual description of each of the one or more segments of audiovisual content comprises: segmenting the audiovisual data into a plurality of audiovisual frames; generating the semantic contextual description for each of the plurality of audiovisual frames; and grouping semantic contextual description for each of the audiovisual frames into the semantic contextual description for the one or more segments based on common content in the audiovisual frames.

31. The computer-implemented method of claim 21 , wherein the prompt from the user comprises a plain language prompt.

32. A computing system, comprising: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising: obtaining audiovisual data descriptive of one or more segments of audiovisual content; generating a semantic contextual description of each of the one or more segments of audiovisual content, the semantic contextual description generated based on one or more content understanding models; providing the semantic contextual description to a user; receiving a prompt from the user comprising one or more user-requested modifications to the semantic contextual description; providing the prompt from the user and the semantic contextual description to the one or more content understanding models; and receiving a second semantic contextual description from the one or more content understanding models, the second semantic contextual description reflecting the one or more user-requested modifications.

33. The computing system of claim 32, wherein the one or more content understanding models comprise a large language model (LLM).

34. The computing system of claim 32, wherein the semantic contextual description comprises text data descriptive of the audiovisual content of the audiovisual data.

35. The computing system of claim 32, wherein the one or more content understanding models comprise a speech-to-text model.

36. The computing system of claim 32, wherein the one or more content understanding models comprise a visual descriptor model.

37. The computing system of claim 32, wherein the one or more content understanding models comprise an ambience descriptor model.

38. The computing system of claim 32, further comprising generating, by the computing system, second audiovisual data based on the audiovisual data and the second semantic contextual description.

39. The computing system of claim 32, wherein generating, by the computing system, the semantic contextual description of each of the one or more segments of audiovisual content comprises: segmenting the audiovisual data into a plurality of audiovisual frames; generating the semantic contextual description for each of the plurality of audiovisual frames; and grouping semantic contextual description for each of the audiovisual frames into the semantic contextual description for the one or more segments based on common content in the audiovisual frames.

40. One or more non- transitory, computer-readable media storing instructions that, when implemented, cause one or more processors to perform operations, the operations comprising: obtaining audiovisual data descriptive of one or more segments of audiovisual content; generating a semantic contextual description of each of the one or more segments of audiovisual content, the semantic contextual description generated based on one or more content understanding models; providing the semantic contextual description to a user; receiving a prompt from the user comprising one or more user-requested modifications to the semantic contextual description; providing the prompt from the user and the semantic contextual description to the one or more content understanding models; and receiving a second semantic contextual description from the one or more content understanding models, the second semantic contextual description reflecting the one or more user-requested modifications.

41. A computer-implemented method for editing audiovisual data, the method comprising: obtaining, by a computing system, original audiovisual data associated with audiovisual content; processing the original audiovisual data, using an automatic speech recognition model, to generate raw transcnpt data of the audiovisual content; processing the raw transcript, using a transcription generation model, to generate an edited transcript data, the edited transcript data having disfluency labeled data removed from the raw transcript; and processing, using an editing model, the edited transcript data and the original audiovisual data to generate modified audiovisual data.

42. The computer-implemented method of claim 41, wherein the edited transcript data is generated using the transcript generation model, wherein the transcript generation model comprises a disfluency detection model, and wherein disfluency labeled data is removed from the raw transcript data to generate the edited transcript data.

43. The computer-implemented method of claim 41, wherein the transcript generation model comprises a speech-to-text model.

44. The computer-implemented method of claim 41, wherein the transcript generation model comprises a visual descriptor model.

45. The computer-implemented method of claim 41, wherein the transcript generation model comprises an ambience descriptor model.

46. The computer-implemented method of claim 41, wherein the editing model comprises a large language model (LLM).

47. The computer-implemented method of claim 41, wherein the edited transcript data comprises text data descriptive of the audiovisual content of the audiovisual data, wherein the text data includes disfluency labeled data, speech data, visual descriptor data, and ambience descriptor data.

48. The computer-implemented method of claim 41, further comprising: receiving a prompt from a user comprising a user-requested modification to the edited transcript data, wherein the user-requested modification comprises a request to modify content of the audiovisual content; and wherein the modified audiovisual data is generated, using the editing model, by processing the edited transcript data, the user-requested modification, and the original audiovisual data.

49. The computer-implemented method of claim 48, wherein the user-requested modification further includes removal of at least a segment of the audiovisual content.

50. The computer-implemented method of claim 48, wherein the user-requested modification includes shortening the audiovisual content, and wherein the editing model automatically removes a segment of audiovisual content.

51. The computer-implemented method of claim 48, wherein the prompt from the user comprises a plain language prompt.

Images