Methods for improving media content editing architecture, computing devices, computing systems, computer-readable media, and computer programs

Abstract

An example of a system evolution architecture for improving a media content editing system is provided. One aspect includes a method for improving a media content editing architecture, which includes editing media content using a backend tool service including a prompt pool for multiple editing tools and multiple application programming interfaces, and a large language model, publishing the edited media content, storing contextual information related to the editing of the media content, and improving the media content editing architecture using the stored contextual information.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the priority of U.S. Application No. 18 / 346,737, filed on July 3, 2023, with the invention title "System Evolutionary Architecture for Improving Media Content Editing System", and the disclosure content of the application is incorporated herein by reference in its entirety.

Background Art

[0002] For better viewer engagement, usually, raw media content in its original recording format is edited before publication to enhance its appeal. Editing media content (such as images, audio, video, and other modalities) is usually provided in the form of editing tools and involves the use of software with editing capabilities. Editing of media content can include various operations and modifications. For example, in the context of video editing, editing can include trimming segments, changing the order of segments, adjusting the playback speed, embedding content such as special effects and caption text, adjusting audio, cropping, etc. Furthermore, the use of powerful editing software enables a non - linear editing (NLE) system where multiple edits to raw media content are performed in a non - destructive process and the original data can be restored, i.e., the edits can be reversed.

Summary of the Invention

Means for Solving the Problems

[0003] An example of a system evolution architecture for improving a media content editing system is provided. One aspect includes a method for improving a media content editing architecture, which includes editing media content using a backend tool service including a prompt pool for multiple editing tools and multiple application programming interfaces, and a large language model, publishing the edited media content, storing contextual information related to the editing of the media content, and improving the media content editing architecture using the stored contextual information.

[0004] This summary is provided to present a simplified excerpt of the concept, which will be further described in the embodiments for carrying out the invention described below. This summary is not intended to identify the main or basic features of the claimed subject matter, nor to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to an implementation that solves any or all of the defects described in any part of this disclosure. [Brief explanation of the drawing]

[0005] [Figure 1] This is a block diagram model illustrating a typical pipeline and various components of an exemplary technical architecture for realizing a media content editing application.

[0006] [Figure 2] This block diagram model shows an exemplary backend tool service that provides editing tools and editing capabilities, which can be realized in the typical pipeline described in Figure 1.

[0007] [Figure 3]This block diagram model shows an example of using context memory 302 in a media content editing architecture, which is achievable in the typical pipeline described in Figure 1.

[0008] [Figure 4] This block diagram model illustrates an exemplary system evolution and refinement application for a media content editing architecture, achievable within the typical pipeline described in Figure 1.

[0009] [Figure 5] This block diagram model shows an exemplary media content editing model architecture with system evolution and refinement processes, providing a detailed illustration of the typical pipeline described in Figure 1.

[0010] [Figure 6] This flowchart illustrates an exemplary method for a media content editing process using machine learning technology, which can be implemented using the technical architecture shown in Figure 1.

[0011] [Figure 7] This flowchart shows an exemplary method for improving a media content editing architecture, which can be achieved using the technical architecture shown in Figure 1.

[0012] [Figure 8] This figure schematically illustrates a non-limiting embodiment of a computing system capable of implementing one or more of the methods and processes described above. [Modes for carrying out the invention]

[0013] Media content editing software, capable of providing powerful editing tools, is widely available for both commercial and personal use. Typically, content editing software involves the use of a user interface (UI) with various sections, menus, buttons, etc., for navigating and selecting desired editing tools. These technologies have evolved over time, offering a vast array of tools for performing numerous editing tasks. However, software with more powerful editing capabilities and features naturally becomes more complex. As a result, many features remain unexplored by the typical user. Complex UI navigation, a lack of knowledge about the software's capabilities, and the difficulty in utilizing these capabilities can all contribute to underutilization of editing software. For example, a typical user of editing software may not recognize or be able to use specific tools or features of the software necessary to perform their desired editing.

[0014] In light of the above observations, a media content editing architecture using machine learning techniques is provided. The machine learning-based architecture can be configured in various ways to provide an intuitive media content editing application. Such an application may be configured to receive editing requests from a user and perform one or more desired edits on the media content provided by the user. In some implementations, the media content to be edited is generated by the application. Editing requests may be provided in text form, and machine learning techniques and natural language processing can be applied to convert the editing requests into one or more edits. Edits may be performed, and rendering results may be provided to the user for evaluation. In some implementations, the editing process is performed as a non-linear editing (NLE) process. Such implementations allow for better utilization of the architecture's editing capabilities in a more flexible manner. The user may revert edits or provide another editing request. The process can be repeated and continued until the user is satisfied, at which point the user publishes the edited media content.

[0015] Various machine learning techniques, such as deep learning models, can be applied. In some implementations, the media content editing architecture includes a large language model (LLM) for analyzing and interpreting user input and predicting one or more editing actions to be performed. Inference prediction can be performed by receiving user text input and responding with dialog replies based on conversational text interaction with the user. The media content editing architecture may further include a prompt manager that provides prompts in response to editing requests. Prompts may be obtained from a prompt database. The prompt manager then generates a list of instructions or actions corresponding to the edits to be performed by feeding the user's request into the provided prompts and performing inference predictions using the LLM. The LLM agent may be further configured to perform the above actions for editing media content. To perform editing, the LLM agent utilizes a register database of available editing tools that the agent can access. The database may be linked to available editing tools and their associated application programming interfaces (APIs) that the LLM agent can use to perform editing actions.

[0016] In some implementations, the media content editing architecture is configured to have a system evolution process that trains and improves the architecture. For example, the LLM and / or prompt database may be improved based on the operation history and edited media content. The media content editing architecture may be configured to store and remember conversation history and / or contextual information (e.g., asset descriptions of edited media content). To prevent dilution of valid samples, the media content editing architecture may be configured to store information about successful submissions (e.g., edited content that is ultimately published by the user). The stored information may be used to improve the media content editing architecture based on a predetermined reward function. For example, the reward function can be determined using various metrics associated with already published edited media content. Such metrics may include those representing the success of the edited media content in terms of viewer engagement. Examples of such metrics include views, comments, likes, shares, etc., associated with published edited media content.

[0017] Next, we will turn to the diagrams to illustrate and describe in more detail a media content editing architecture utilizing machine learning technology. Figure 1 is a block diagram model showing a computing system 100. The block diagram model represents a general pipeline and various components of an exemplary technical architecture for realizing a media content editing application in a client-server environment. The computing system 100 includes a server system 101, which includes multiple server computing devices configured to realize a social media network platform by executing the illustrated modules and services. The server system 101 is configured to communicate with multiple client computing devices 103, each running a social network client 102, via a computer network N, for example, the Internet. For example, the computing system 100 can realize a short video social media network where users create, publish, share, and engage with short videos. In other realizations, the computing system 100 may be realized as an offline application on a computing device. It will be understood that certain modules shown on the server system may be realized on a client computing device, for example, a backend editing tool. Furthermore, the social network client may be a mobile client of the social media network, an effects editing software program running on a personal computer, or other software.

[0018] The editing process is performed via a dialog-assisted editing interface 104, which includes a dialog interface 106. The user 108 provides the media content 110 to be edited, along with an editing request 112. In some implementations, the media content 110 is generated by a media content editing application in response to the user's request. For example, the media content can be generated using generative machine learning techniques. The media content 110 may have various modalities. For example, the media content 110 may be an image, audio recording, video, etc. The media content 110 may be displayed via the dialog-assisted editing interface 104. Furthermore, edits performed on the media content 110 during the editing process can be displayed to the user 108 via the dialog-assisted editing interface 104, allowing the user 108 to evaluate their next steps.

[0019] An edit request 112 is provided to the prompt manager module 114. In some implementations, the edit request 112 is provided in the form of text input. Accordingly, the prompt manager module 114 queries the prompt pool 116 to obtain a prompt 118. In some implementations, the obtained prompt 118 is selected from the prompt pool 118 based on the edit request. The prompt pool 116 may include a database of predefined prompts, each of which may be associated with the editing capabilities of the computing system 100. For example, a prompt may include a basic description of a given tool, typical questions associated with the tool, a defined input format to the tool, and / or possible intermediate steps when using the tool. The use of the prompt pool 116 offers several advantages. One advantage is the standardization of input. Another advantage is the flexibility in extending the editing toolset. For example, if a new tool is added to the editing capabilities of the computing system, a corresponding prompt may be added to the prompt pool 116.

[0020] The Prompt Manager Module 114 fills the obtained prompt 118 with the editing request 112 and passes it to the LLM Agent 120. In some implementations, the computing system 100 includes a Content Asset Analyzer 121 for processing the media content 110 to generate metadata that can be provided as input to the LLM Agent 120. For example, the Content Asset Analyzer 121 may preprocess video content to extract individual frames, analyze the visual and audio content of the video content, and generate video metadata that may include text descriptions of the analyzed visual and audio content, recognized entities, timestamps for key events, and video captions of the video content.

[0021] The LLM agent 120 includes an LLM prediction module 122 that utilizes the LLM 124 to perform inference predictions on received inputs. The LLM 124 can be implemented as a language model consisting of a trained neural network with numerous parameters. The LLM 124 can be trained as a general-purpose model or for a limited range of tasks. For example, a media content editing architecture can be implemented with a single general-purpose trained LLM, or with multiple LLMs, each trained for different tasks. In some implementations, a set of LLMs, each trained for a specific range of tasks, is provided, and the LLM agent 120 selects the LLM to use based on the received prompt 118. By utilizing the prompt 118 along with the user's editing request 112, structure and context are provided to the input to the LLM 124. Therefore, since the input is predictable to some extent from a structural standpoint, the LLM 124 is able to provide more accurate inference predictions. The LLM agent 120 may be configured to provide an interactive text conversation with the user 108, in which a dialog response 126 is generated using the LLM 124 and returned to the user 108 via the dialog interface 106. The user 108 can then provide new text input to advance the conversation. The conversation continues until the LLM agent 120 decides to end the conversation, which may be based on new text input and / or the current round number in the conversation. Once the conversation is ended, the LLM prediction module 122 generates an inference prediction using the LLM 124 based on the received text input.

[0022] The LLM agent 120 includes an action planning and execution module 128 that analyzes inference predictions to create a list of editing actions. The possible editing actions can be selected from a tool database 130 that lists the editing tools available to the computing system 100 when editing media content 110. The tool database 130 is provided in a backend service 132 that includes tools 134 and associated APIs 136. Tools for editing media content can include, but are not limited to, tools for adding, deleting, and / or modifying content of various modalities, such as text, images, videos, audio, etc. For example, a tool can be implemented to embed a recording into video content. In some implementations, the added content is created using a generation process.

[0023] The action planning and execution module 128 executes the list of editing actions using appropriate API 136 calls to the tools 134 necessary to execute the editing actions. The editing actions are executed on the media content 110 provided by the user 108, and the edited media content 138 is returned to the user 108 via the dialog-assisted editing interface 104. In the dialog-assisted editing interface 104, the rendering of the edited media content 138 is displayed to the user, and the user 108 views it and decides on the next action. For example, the user 108 may decide to undo the executed edit, provide a new edit request 112 for additional editing, or publish (140) the edited media content 138. Upon publishing (140), a copy of the edited media content 138 can be stored on the content server 142. In the example shown in FIG. 1, the published media content 144 is provided on the social network client 102 for viewing by other users 146 of the platform.

[0024] In some implementations, the media content editing architecture includes a system evolution process that improves (refines) the ability to propose and / or execute actions / edits more efficiently. Various types of feedback are available in the system evolution process. For example, direct user feedback (e.g., user 108 can provide feedback in the form of a rating system where the prompts and / or tools used when performing the edit are the cause of effectiveness). Another example of feedback involves the use of conversation history and / or contextual information of a successful submission (e.g., published media content 144). Different reward functions may be used to determine the impact of a given refinement iteration. In the example shown in Figure 1, the computing system 100 includes a platform audience engagement aggregation module 148 for providing information about audience engagement metrics regarding published media content 144. Exemplary metrics include views / listenings, comments, shares, likes, etc. Higher audience engagement metrics suggest more "successful" edited media content. Therefore, more weight can be given to the information used in the refinement process associated with published media content that has higher audience engagement metrics. For example, when a predetermined audience engagement metric threshold (e.g., a predetermined number of video views within a predetermined time frame) is reached, a refinement process may be performed on the published media content that has reached the threshold.

[0025] The refinement process can be performed on various modules within the architecture. In the example in Figure 1, the computing system 100 includes a prompt refinement module 150 for improving the prompt pool 116. The computing system 100 further includes an LLM fine-tuning module 152 for improving the LLM 124. Conversation history and / or contextual information of a given published media content can be used to improve the prompt pool 116 and / or LLM 124. For example, it is not practical to provide all editing options for a given prompt. Therefore, generally, one set of available options is provided for a given prompt. Refinement of the prompt pool can affect the set of options provided to the user. By using a high audience engagement metric as a surrogate metric for the success of the edited media content, the conversation history and / or contextual information related to the editing of the media content can be used to improve the options provided in a given prompt so that more "popular" options are provided. As a more specific example, in response to a user request to embed music in video content, the prompt may initially include options for different music genres. The prompts may be improved to include options for the above-mentioned popular genres, based on published content that demonstrates a high audience engagement metric, once they have been edited to embed more popular genres of music. In this way, the computing system 100 can be continuously updated to better respond to user editing requests.

[0026] Figure 2 is a block diagram showing an example configuration of the computing system 100 of Figure 1. Figure 2 shows an exemplary backend tool service 132 for providing editing tools and capabilities available in the computing system 100. The exemplary backend tool service 132 provides backend support and functionality that the LLM agent can use to perform editing on media content (for example, the action plan and execution module 128 may use the backend tool service 132 to perform editing on media content 110).

[0027] The backend tool service 132 includes a repository of available editing tools / capabilities for a media content editing architecture. In some implementations, tools are located and organized within groups. In another embodiment, tools are organized within a multi-level hierarchy. Such an organization method allows for conversational interaction that presents the user with a practical number of options for a given selection. For example, instead of listing all available tools and letting the user choose, providing groupings first can narrow down the user's desired editing and provide more appropriate context.

[0028] In the example shown in Figure 2, tool 134 is organized into group 202. For example, music recommendation tool 134A and filter recommendation tool 134N are shown as organized under recommendation group 202. Other groups and classifications shown include understanding, describing, artificial intelligence (AI) generation, search and matching, localization, structural analysis, AI modification, and evaluation. Each grouping includes various editing capabilities across various modalities, e.g., images, videos, music, text, and audio. Exemplary capabilities for understanding groupings may include content embedding and tagging. Exemplary capabilities for localization groupings may include object detection, event detection, character recognition, object segmentation, event / scene detection, and beat / chorus / beginning detection. Exemplary capabilities for describing groupings may include image captioning, video captioning, title generation, and text summarization. Exemplary capabilities for structural analysis groupings may include slicing (shot boundary) and highlight detection. Exemplary capabilities for AI-generated grouping may include various generation processes for content generation, such as image generation, video generation, music generation, and video script generation. Exemplary capabilities for AI-modified grouping may include trimming, volume adjustment, voice modification, noise reduction, super-resolution, cropping, background removal, tone mapping, inpainting, video-audio synchronization, and speed curving. Exemplary capabilities for search and matching grouping may include material search and material replacement. Exemplary capabilities for recommendation grouping may include recommendation and application of various content, such as filters, music, titles, narrative speech, animations, special effects, stickers, and text (including different fonts, styles, animations, and positions). Exemplary capabilities for evaluation grouping may include image quality, video quality, and music quality. For ease of understanding, the backend tool service 132 may include any number of groupings 202 using any classification scheme.Furthermore, each grouping 202 may contain any number of tools 134, and these tools 134 may be further classified into subgroupings in some examples.

[0029] Each tool 134 includes information describing a callback API 136 that can be used by the LLM agent to invoke when the editing tool performs edits on the media content. The collection of callback APIs 136 is aggregated in a tool API pool 130, which serves as a repository accessible by the LLM agent. For example, as shown in Figure 1, the action plan and execution module 128 utilizes the tool API pool 130 to execute a list of editing actions it generates in order to form the edited media content 138.

[0030] For each tool 134, a corresponding prompt 204 is generated. The collection of prompts 204 is aggregated in a prompt pool 116, which in some examples may be accessed by a prompt manager during prompt retrieval. Prompts 204 can have various formats. In some implementations, a prompt 204 for a given tool includes a basic description of the tool, typical questions related to the tool, a predefined input format for the tool, and / or intermediate steps when using the tool. The backend tool service 132 may be dynamically implemented to have the ability to add and remove tools 134. When a new tool 134 is added, a corresponding prompt 204 may be generated, added to the backend tool service 132, and consequently added to the prompt pool 116.

[0031] Figure 3 is a block diagram showing an example configuration of the computing system 100 in Figure 1. Figure 3 illustrates an example of the use of context memory 302 within the media content editing architecture available in the computing system 100. Directed connections are illustrated to show the relationships between components related to context memory 302. One direct method for editing media content 110 involves a direct command 303 given by the user via a dialog-assisted editing interface 104. The direct command 303 describes a specific editing action desired by the user and is formatted so that the media content editing architecture can understand the given command without using LLM or LLM agent 120. Thus, the direct command 303 allows the user to directly invoke editing tools from the backend tool service 132 to perform edits on the media content 110. For more advanced unstructured queries, context memory 302 can be used to store context information and guide the editing process.

[0032] The context memory 302 can contain storage for various contextual information that the media content editing architecture can use for various purposes. For example, during media content editing by the action plan and execution module 128, an edit draft history 304 can be compiled based on a list of nonlinear edits and associated editing tools. The edit draft history 304 may include steps and edits for rendering the edited media content to the user via the dialog-assisted editing interface 304. The context memory 302 further comprises an editing context 306 that provides context to the editing capabilities and tools provided by the backend tool service 132.

[0033] Conversational interactions between the user and the LLM can also be memorized. In the example shown in Figure 3, the conversation history 308 is stored in the context memory 302. For example, a dialogue can be memorized when conversational input is provided to the LLM agent 120 and when the LLM prediction module 122 generates a dialogue response 126 using the LLM. The conversation history 308 can be used for various purposes. During the editing process, the conversation history 308 can provide information to the media content editing architecture to determine how many edits have been performed. In some implementations, the media content editing architecture is configured to suggest publishing the edited content 110 after a certain amount of conversational exchange and / or multiple rounds of editing. Another use involves prompt suggestions based on interactions in previous conversations. For example, the conversation history 308 may store previous interactions in which the user rejected suggested edits, and the media content editing architecture may be configured to be less likely to provide relevant prompts.

[0034] In some implementations, the conversation history 308 and / or edit draft history 304 may be used for training or refinement of the media content editing architecture, and the refinement may include refinement of the prompt pool and / or LLM. The conversation history 308 and / or edit draft history 304 may be stored for each user submission, and their contents may be used to improve the prompt pool and / or LLM. For example, published edited media content with a high audience engagement metric may be considered a training sample for the refinement process. The conversation history 308 and / or edit draft history 304 of the published edited media content may be used to improve the prompt pool and / or LLM so that the prompts and dialogue responses 126 associated with the published edited media content are more likely to appear in future interactions.

[0035] Figure 4 is a block diagram showing an example configuration of the computing system 100 in Figure 1. Figure 4 illustrates exemplary system evolution and refinement applications for a media content editing architecture usable in the computing system 100. In the example shown in Figure 4, exemplary system evolution and refinement processes are performed for the backend tool service 132 and LLM 124. Various processes can be used for the refinement of the media content editing architecture. In some implementations, reinforcement learning algorithms, such as reinforcement learning from human feedback (RLHF) and proximal policy optimization (PPO), are implemented. For example, information recorded from a successful editing process 402 can be used to improve the prompts provided by LLM 124 and the backend tool service 132 to provide more relevant prompts and responses in future interactions with the user. Records of successful editing processes can be provided at various stages of the editing process. Various information, such as conversation history and contextual information (e.g., asset descriptions), may be recorded. Such prompt / query and response pairs within the information can be used as training samples for the refinement process. The conversation history may include the results of both successful and less successful conversations. For example, as illustrated in Figure 3, the context memory 302 may be implemented to store various information about the editing process, such as the conversation history 308, the edit draft history 304, the editing context 306, etc.

[0036] A "successful" editing process can be defined in various ways. In some implementations, the editing process is considered successful when the edited media content is published. At that point, information related to the editing process, such as conversation history 308 and edit draft history 304, is recorded. In other implementations, all editing process interactions with the user are recorded. However, this can generate a huge amount of unnecessary data that has little effect on whether prompts and tool suggestions were effective. In other implementations, the editing process of published edited media content is considered successful when a predetermined audience engagement threshold is reached.

[0037] In the example shown in Figure 4, the information recorded for a given successful editing process is organized into user-accepted interactions 404 and user-rejected interactions 406. Such interactions may include user responses to proposed editing and tool options provided by the prompt manager and / or LLM agent. Model 400 includes an editing experience pool 408 that aggregates the records of the successful editing process 402, including the user-accepted and rejected interactions 404, 406. The aggregated information in the editing experience pool 408 is available to the prompt refinement module 150 to improve the backend tool service 132. More specifically, the editing experience pool 408 can be used to improve prompts provided by the backend tool service 132. For example, prompts can be modified in accordance with information in the editing experience pool 408 that describes efficient and inefficient prompts, which can be associated with user-accepted interactions 404 and user-rejected interactions 406, respectively. User-accepted interaction 404 can provide context suggesting that the user-accepted prompt is more likely to lead to a more successful editing process. Therefore, similar prompts may be configured to be suggested more frequently in future interactions. Similarly, prompts associated with user-rejected interaction 406 may be modified accordingly or configured to be suggested less frequently in future interactions.

[0038] Furthermore, aggregated information within the editing experience pool 408 may be used by the LLM fine-tuning module 152 to improve the LLM 124. Similar to the prompt refinement module 150, the LLM fine-tuning module 152 can improve the LLM 124 by utilizing information within the editing experience pool that describes efficient and inefficient interactions as positive and negative reinforcement data, respectively. In some implementations, the reward function is implemented to determine the extent to which the information within the editing experience pool influences the refinement process. Various reward models are possible. In the example shown in Figure 4, online performance data is used as a reward model 410 to improve the LLM 124. Online performance data for published edited media content can be quantified using various audience engagement metrics and indicators, such as views, likes, shares, and comments. A platform audience engagement aggregation module 148, for example illustrated and described in Figure 1, may be used to aggregate relevant audience engagement indicators for the published edited media content from the hosting service of the published edited media content. Such data can be fed into the reward model 410 to determine the weights of the training samples (information within the edit experience pool 408) in the refinement process. Figure 4 shows how the evolution and refinement system uses online performance data as a reward model for the refinement of LLM 124, although such a model is also available for the refinement of prompts in the backend tool service 132.

[0039] Figure 5 is a block diagram illustrating an exemplary media content editing model architecture with system evolution and refinement processes, usable with the computer system 100 of Figure 1. Figure 5 provides a detailed illustration of the pipeline flow of a conversational, nonlinear editing process using the exemplary content editing model architecture. The process begins with a user 108 interacting with a dialogue-assisted editing interface 104 and providing media content 110 to be edited. The media content 110 may be any modality, including images, audio, video, etc. In some implementations, the media content 110 is generated by the exemplary media content editing model architecture via a generative AI process. The dialogue-assisted editing interface 104 can be implemented on any computing device. In some implementations, the dialogue-assisted editing interface 104 is provided within a social network client, for example, the social network client 102 shown in Figure 1. The social network client may include various social networking platforms, such as short video social media platforms, as described above.

[0040] The dialog-assisted editing interface 104 provides an interface that allows user 108 to view media content 110 during the editing process, such as rendering the results of selected edits. Furthermore, the dialog-assisted editing interface 104 includes a dialog interface 106 that can send and receive text commands. The editing process includes user 108 providing an edit request 112 using the dialog interface 106. The edit request 112 is provided to the prompt manager module 114. Because the editing capabilities of the exemplary media content editing model architecture may include numerous editing tools, the prompt manager module 114 may be implemented to contribute to structuring and narrowing down the edit requests into a subset of the architecture's editing capabilities. Through prompt engineering, the prompt manager module 114 and the prompt acquisition module 502 operate to acquire prompts from the prompt pool 116. The acquired prompts are typically related to the edit request 112. For example, if the edit request 112 is related to music, the prompt acquisition module 502 can query the prompt pool 116 to acquire prompts related to music. In some implementations, the query provides a set of prompts with similar descriptions that match the edit request 112, and this set of prompts is combined with a fixed prompt about a tool associated with the edit request 112 to form a new prompt.

[0041] Generally, the prompt pool 116 contains at least one prompt corresponding to each editing capability. The prompt pool 116 can be implemented as a dynamic database where prompts can be added, deleted, and modified, thus providing flexibility in extending the architecture's editing toolset. For example, if a new tool is registered in the toolset, a corresponding prompt may also be added to the prompt pool 116. Prompts can be formatted in various ways. In some implementations, a prompt includes a basic description of the tool, typical questions related to the tool, a predefined input format for the tool, and / or possible intermediate steps when using the tool.

[0042] The edit request 112 and the acquired prompt can be supplied to the LLM prediction module 122 of the LLM agent 120, which uses the LLM 124 to perform inference prediction. The LLM agent 120 can be implemented as a text command transmitter / receiver providing conversational interaction with the user 108, and the LLM agent 120 uses the LLM 124 to predict the response to the text input it receives (edit request 112 and prompt). Since prompts are generally predefined, the LLM 124 can output structured results. In some implementations, the LLM 124 is a single general-purpose LLM. In other implementations, the LLM agent 120 can access a repository of multiple LLMs, each trained for one or more specific tasks. In such cases, the choice of which LLM to use may be based on the edit request 112 and / or prompt.

[0043] The LLM agent 120 may be configured to convert the structured results from LLM 24 into a tool execution sequence and inputs for executing the tool. The LLM agent 120 includes an LLM output analysis unit 504 that obtains structured information within the predicted response by analyzing the predicted response from the LLM prediction module 122. The LLM agent 120 further includes an LLM action planning module 128A and an LLM tool execution model 128B. The LLM action planning module 128A and the LLM tool execution model 128B may be implemented similarly to the action planning and execution module 128 in Figure 1. The LLM action planning module 128A may be implemented to plan actions to be performed based on the structured information. Based on the planned actions, the LLM tool execution module 128B forms a toolchain and executes the toolchain using API calls from the tool API pool 130 for the tools in the toolchain. For open-ended questions or complex requests, the LLM agent 120 can perform self-search using the LLM 124, specifically by using the self-search module 506 and the tool execution chain module 508, respectively, generating multiple intermediate steps. At each step, the LLM 124 can gradually approach the final answer by using the search or subsequent questions. Text of the conversational exchange is feasible. For example, the dialog response 126 and subsequent responses may be provided to the user 108 via the dialog interface 106 of the dialog-assisted editing interface 104. In some implementations, the dialog response is stored in a context memory 302 that records the conversation history 308.

[0044] When the toolchain is executed, it uses API calls to the tools within the toolchain to perform editing on the media content 110. The backend tool service 132 provides editing capabilities, performs the editing steps, and stores the above steps in the editing draft history 304 in context memory 302. The edited media content is provided to the user 108 via the dialog-assisted editing interface 104, and the user 108 may decide on a series of actions. For example, the user 108 may decide to revert the edits, provide an additional editing request, or publish (140) the edited media content.

[0045] Model 500 includes a system evolutionary architecture that enables a refinement process for a media content editing architecture. The refinement process can be implemented using components and methods similar to those described in Figure 4. In the example shown in Figure 5, the prompt refinement module 150 and the LLM fine-tuning module 152 are implemented to improve prompts in the backend tool service 132 and LLM 124, respectively. Training data for the refinement process may include various contextual information stored during the editing process. For each submission (a set of interactions with the user 108 about a given media content 110), the conversation history 302 and contextual information, such as an asset description, may be stored in the context memory 302. This conversation history may include the results of both successful and less successful conversations. In some implementations, only successful submissions are saved. A “successful” submission can be defined in various ways. For example, a submission may be considered successful when the edited media content is published (140).

[0046] When the edited media content is published, a record of the successful editing process 402 is obtained. Such a record may include context data stored during the editing process for the edited media content, for example, data stored in context memory 302. In some examples, the context data may be separated into user-adopted interactions 404 and user-rejected interactions 406. Such interactions may include user responses to proposed editing and tool options. The editing experience pool 408 aggregates the context data, which is then used by the prompt refinement module 150 and the LLM fine-tuning module 152, respectively, to improve the backend tool service 132 and LLM 124.

[0047] The reward model 410 can be implemented to assign different weights to the training data. Rewards can be based on various criteria. In the example shown in Figure 5, online performance data in the form of an audience engagement metric is used as the reward function. Higher audience engagement indicates a higher reward for the training data (contextual data) that generated the published edited content. The audience engagement metric may include various metrics related to the online performance data of the published edited media content. Exemplary metrics include views, likes, shares, and comments. The platform audience engagement aggregation module 148, for example illustrated and described in Figure 1, can be used to aggregate relevant audience engagement metrics for the published edited media content from the hosting service of the published edited media content. In some implementations, the reward model 410 is also applied in a similar manner to the prompt refinement module 150.

[0048] Figure 6 is a flowchart illustrating an exemplary method 600 for a media content editing process utilizing machine learning techniques. Such a method may be implemented on a media content editing architecture, for example, one illustrated and described in Figure 5. In step 602, method 600 includes receiving media content from a user. Various types of media content and modalities are available. For example, media content may be images, audio recordings, or videos. Media content may be provided by the user, for example, through an upload process. In some implementations, media content is provided by a generative AI process.

[0049] In step 604, method 600 includes receiving an edit request from a user regarding the media content. The edit request may be received from a user by using a dialog-assisted editing interface. Typically, the edit request is received in the form of text input. The edit request may be received using a prompt manager module. The edit request may include a request to revert previous edits made to the media content. In some implementations, the edit request may be a direct command in a structured format that allows direct access to the editing tools of the media content editing architecture.

[0050] In step 606, method 600 includes editing media content based on an edit request to produce edited media content. Editing media content can be performed using various processes. Substeps 606A to 606C describe one such process. In substep 606A, method 600 includes obtaining a prompt from a prompt pool. A prompt can be obtained using a prompt manager module. The prompt pool may contain multiple prompts, each corresponding to at least one editing tool.

[0051] In substep 606B, method 600 includes parsing the acquired prompts and the edit requests in a large language model to generate one or more edit actions to be performed on the media content. An LLM agent may be used to receive the input and feed the input to the large language model. The use of prompts can enable more structured input, which in turn allows the large language model to provide a more consistent response. The large language model may be configured to parse the input and generate one or more edit actions in the form of an action toollist.

[0052] In substep 606C, method 600 includes performing one or more editing actions on media content to produce edited media content. Performing editing actions may include using API calls to corresponding editing tools. APIs may be obtained from a tool API pool.

[0053] In step 608, method 600 optionally includes publishing the edited media content. The edited media content may be published on various platforms. For example, the edited media content may be published on a short video social media network.

[0054] Figure 7 is a flowchart illustrating an exemplary method 700 for improving a media content editing architecture. Improving the media content editing architecture can be achieved using a system evolutionary architecture, for example, one illustrated and described in Figure 4. In step 702, method 700 includes editing media content using a media content editing architecture, for example, one illustrated and described in Figure 5. The method described in Figure 6 may be used to edit the media content. Various types of media content and modalities are available. For example, the media content may be images, audio recordings, or videos. The media content editing architecture may include a large language model and backend tool services. The backend tool services may include a prompt pool.

[0055] In step 704, method 700 includes publishing the edited media content. The edited media content may be published on various platforms. For example, the edited media content may be published on a short video social media network.

[0056] In step 706, method 700 includes storing contextual information related to the editing of media content. Examples of contextual information include conversational history, editing context, and editing draft history. In some implementations, contextual information includes an asset description of the edited media content. In some implementations, contextual information is stored in context memory. Contextual information can be used for various purposes. During the editing process, contextual information is aware of past actions in the editing process that can influence the dialog responses of the media content editing architecture. For example, if contextual information includes a conversational history in which a user rejected a given proposed edit, the media content editing architecture may be configured not to propose the above edit for a given editing process. Another use of contextual information includes refining the media content editing architecture.

[0057] In step 708, method 700 includes improving the media content editing architecture using stored contextual information. Improving the media content editing architecture may include improving the prompt pool and / or the large-scale language model. In some implementations, the stored contextual information includes conversation history categorized into user-accepted and user-rejected interactions, and improving the media content editing architecture includes improving the prompt pool based on user-accepted and user-rejected interactions. For example, prompts in the prompt pool may be improved to suggest relevant editing actions corresponding to user-accepted interactions rather than user-rejected interactions. The refinement process may include using an audience engagement metric associated with the published edited media content as a reward function. Exemplary audience engagement metrics include views, likes, shares, and comments. In some implementations, the refinement process is performed when a threshold of a given audience engagement metric (e.g., a given number of video views within a given time frame) is reached.

[0058] In some embodiments, the methods and processes described herein may be linked to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer application programs or services, application programming interfaces, libraries, and / or other computer program products.

[0059] Figure 8 schematically illustrates a non-limiting embodiment of a computing system 800 capable of implementing one or more of the methods and processes described above. The computing system 800 is shown in a simplified form. The computing system 800 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing, game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, and wearable computing devices, such as smart watches and head-mounted augmented reality devices.

[0060] A media content editing architecture can be implemented to provide general users with intuitive editing tools and experiences. By using LLM in combination with a wide range of editing tools, users can perform powerful editing on media content without requiring advanced software knowledge. Such an architecture can receive user input as unstructured text, predict desired editing requests using prompts and natural language processing techniques, and perform these predictions using a pool of available editing tools. Another implementation may involve refining such a technical architecture. By utilizing online performance data of published edited content, the system can evolve and improve itself without requiring the costly and labor-intensive training process of traditional LLM models.

[0061] The computing system 800 comprises a logical processor 802, a volatile memory 804, and a non-volatile storage device 806. The computing system 800 may optionally include a display subsystem 808, an input subsystem 810, a communication subsystem 812, and / or other components not shown in Figure 8.

[0062] The logical processor 802 includes one or more physical devices configured to execute instructions. For example, the logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or achieve desired results.

[0063] A logic processor may include one or more physical processors (hardware) configured to execute software instructions. Furthermore, or alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute logic or firmware instructions implemented in hardware. The processor of logic processor 802 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of the logic processor may optionally be distributed across two or more separate devices located remotely and / or configured for coordinated processing. Embodiments of the logic processor may be virtualized and executed by computing devices connected to a remotely accessible network configured in a cloud computing configuration. In such cases, it should be understood that these virtualized embodiments run on different physical logic processors on various different machines.

[0064] The non-volatile storage device 806 includes one or more physical devices configured to hold instructions executable by a logical processor in order to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 806 may be transformed, for example, to hold different data.

[0065] The non-volatile storage device 806 may include removable and / or built-in physical devices. The non-volatile storage device 806 may include optical memory (e.g., CD, DVD, HD-DVD, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk® drive, tape drive, MRAM, etc.), or other mass storage technology. The non-volatile storage device 806 may include non-volatile, dynamic, static, read / write, read-only, sequential access, position-addressable, file-addressable, and / or content-addressable devices. It will be understood that the non-volatile storage device 806 is configured to retain instructions even when power to the non-volatile storage device 806 is cut off.

[0066] The volatile memory 804 may include a physical device that provides random access memory. The volatile memory 804 is typically used by the logical processor 802 to temporarily store information during the processing of software instructions. It will be understood that if power to the volatile memory 804 is cut off, the volatile memory 804 will typically no longer store instructions.

[0067] Embodiments of the logic processor 802, volatile memory 804, and non-volatile storage device 806 may be integrated into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), integrated circuits for specific programs and applications (PASICs / ASICs), standard products for specific programs and applications (PSSPs / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0068] The terms “module,” “program,” and “engine” may be used to describe a form of computing system 800 that is generally implemented in software to perform a specific function, which involves a conversion process that specially configures the processor to perform a certain function using a portion of volatile memory. Thus, a module, program, or engine may be instantiated using a portion of volatile memory 804 via a logical processor 802 that executes instructions held by a non-volatile memory 806. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated from different applications, services, code block, object, routine, API, function, etc. The terms “module,” “program,” and “engine” may include individuals or groups such as executable files, data files, libraries, drivers, scripts, database records, etc.

[0069] The display subsystem 808, if included, may be used to present a visual representation of the data held by the non-volatile memory 806. This visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein modify the data held by the non-volatile memory and transform the state of the non-volatile memory, the state of the display subsystem 808 may also be transformed to visually represent the changes in the underlying data. The display subsystem 808 may include one or more display devices utilizing substantially any type of technology. Such display devices may be combined with the logical processor 802, volatile memory 804 and / or non-volatile memory 806 within a shared enclosure, or such display devices may be peripheral display devices.

[0070] The input subsystem 810 may include, if included, one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller, and may be connected to them. In some embodiments, the input subsystem may include, and may be connected to, selected natural user input (NUI) components. These components may be integrated or peripheral, and the transmission and / or processing of input actions may be handled onboard or offboard. Exemplary NUI components may include a microphone for speech and / or voice recognition, an infrared camera, color camera, stereo camera, and / or depth camera for machine vision and / or gesture recognition, a head tracker, eye tracker, accelerometer and / or gyroscope, and / or any other suitable sensors for motion detection and / or intent recognition.

[0071] The communication subsystem 812, if included, may be configured to connect the various computing devices described herein to each other or to other devices in a communicative manner. The communication subsystem 812 may include wired and / or wireless communication devices compatible with one or more different communication protocols. In a non-limiting example, the communication subsystem may be configured to communicate over a wireless telephone network, or over a wired or wireless local or wide area network. In some embodiments, the communication subsystem may allow the computing system 800 to send and receive messages with other devices over a network such as the Internet.

[0072] The following paragraphs provide further explanation of the subject matter of this disclosure. One embodiment provides a method for improving a media content editing architecture, the method comprising editing media content, publishing the edited media content, storing contextual information relating to the editing of the media content, and improving the media content editing architecture using the stored contextual information, the backend tool service including a prompt pool corresponding to a plurality of editing tools and a plurality of application programming interfaces, and a large language model. In this embodiment, further or alternatively, improving the media content editing architecture includes improving the large language model and the prompt pool. In this embodiment, further or alternatively, the stored contextual information includes a conversation history categorized into user-accepted and user-rejected interactions, and improving the media content editing architecture includes improving the prompt pool based on the user-accepted and user-rejected interactions. In this embodiment, further or alternatively, improving the media content editing architecture includes improving the large language model using the contextual information and a reward function. In this embodiment, further or alternatively, the reward function is based on one or more audience engagement metrics associated with the published edited media content. In this embodiment, further or alternatively, the one or more audience engagement metrics include one or more of the following metrics: views, likes, shares, or comments. In this embodiment, further or alternatively, improving the large language model includes improving the large language model using the contextual information when the one or more audience engagement metrics reach a predetermined threshold. In this embodiment, further or alternatively, the predetermined threshold includes reaching a predetermined number of views within a predetermined period from the publication of the edited media content.In this embodiment, the contextual information further or alternatively includes one or more of conversation history, editing context, or editing draft history. In this embodiment, further or alternatively, the media content is published on a short-form social media platform. Furthermore, in this embodiment, a non-temporary computer-readable medium is provided which, when executed by a computing device, causes the computing device to implement the method described herein.

[0073] Another embodiment provides a computing device for improving a media content editing architecture, the computing device comprising a processor and memory, the processor being configured to edit media content, publish the edited media content, store context information related to the editing of the media content, and improve the media content editing architecture using the stored context information, by executing a program using a portion of the memory, a backend tool service including a prompt pool corresponding to multiple editing tools and multiple application programming interfaces, and a large language model. In this embodiment, further or alternatively, the stored context information includes a conversation history classified into user-accepted and user-rejected interactions, and improving the media content editing architecture includes improving the prompt pool based on the user-accepted and user-rejected interactions. In this embodiment, further or alternatively, improving the media content editing architecture includes improving the large language model using the context information and a reward function. In this embodiment, further or alternatively, the reward function is based on one or more audience engagement metrics associated with the published edited media content, wherein the one or more audience engagement metrics include one or more of the following metrics: views, likes, shares, or comments. In this embodiment, further or alternatively, the contextual information includes one or more of the following: conversation history, editing context, or editing draft history.

[0074] Another embodiment provides a computing system for improving a media content editing architecture, the computing system comprising: a social media network application with a dialog-assisted editing interface; memory storing one or more large language models; and a processor, the processor being configured to edit media content using a backend tool service including a prompt pool and application programming interfaces for multiple editing tools, the dialog-assisted editing interface, and the one or more large language models by executing a program using a portion of the memory; publishing the edited media content using the social media network application; storing context information related to the editing of the media content in the memory; and improving the media content editing architecture using the stored context information. In this embodiment, further or alternatively, the context information includes a conversation history classified into user-accepted and user-rejected interactions, and improving the media content editing architecture includes improving the prompt pool based on the user-accepted and user-rejected interactions. In this embodiment, further or alternatively, improving the media content editing architecture includes improving the one or more large-scale language models using the contextual information and reward function. In this embodiment, further or alternatively, the reward function is based on one or more audience engagement metrics associated with the published edited media content, wherein the one or more audience engagement metrics include one or more of the following metrics: views, likes, shares, or comments.

[0075] It will be understood that the settings and / or approaches described herein are essentially illustrative and numerous variations are possible, and therefore these specific embodiments or examples should not be considered restrictively. The specific routines or methods described herein may represent one or more of any number of processing strategies. For this reason, the various operations illustrated and / or described may be executed in parallel, in any other order, or omitted, in the order illustrated and / or described. Similarly, the order of the processes described above may be changed.

[0076] The subject matter of this disclosure includes novel and non-obvious combinations and subcombinations of the various processes, systems, settings, and other features, functions, operations and / or characteristics disclosed herein, as well as all equivalents thereof.

Claims

1. A method for improving media content editing architecture, Editing media content using a backend tool service that includes a prompt pool supporting multiple editing tools and multiple application programming interfaces, along with a large-scale language model. To publish the aforementioned edited media content, To store contextual information related to the editing of the media content, To improve the media content editing architecture using the stored context information, A method that includes this.

2. Improving the media content editing architecture includes improving the large-scale language model and the prompt pool. The method according to claim 1.

3. The stored contextual information includes conversation history categorized into user-accepted and user-rejected interactions, and improving the media content editing architecture includes improving the prompt pool based on the user-accepted and user-rejected interactions. The method according to claim 1.

4. Improving the media content editing architecture includes improving the large-scale language model using the contextual information and reward function. The method according to claim 1.

5. The aforementioned reward function is based on one or more audience engagement metrics associated with the published edited media content. The method according to claim 4.

6. The aforementioned one or more viewer engagement metrics include one or more metrics of views, likes, shares, or comments. The method according to claim 5.

7. Improving the large-scale language model includes improving the large-scale language model using the context information when one or more audience engagement indicators reach a predetermined threshold. The method according to claim 5.

8. The predetermined threshold includes reaching a predetermined number of views within a predetermined period from the publication of the edited media content. The method according to claim 7.

9. The aforementioned context information includes one or more of the following: conversation history, editing context, or editing draft history. The method according to claim 1.

10. The aforementioned media content will be published on short-form social media platforms. The method according to claim 1.

11. A computing device for improving media content editing architecture, A computing device comprising a processor and memory, wherein the processor executes a program using a portion of the memory, Using a backend tool service that includes a prompt pool supporting multiple editing tools and multiple application programming interfaces, along with a large-scale language model, media content is edited. The aforementioned edited media content will be made public. The system stores contextual information related to the editing of the media content, The system is configured to improve the media content editing architecture using the stored context information. Computing device.

12. The stored contextual information includes conversation history categorized into user-accepted and user-rejected interactions, and improving the media content editing architecture includes improving the prompt pool based on the user-accepted and user-rejected interactions. The computing device according to claim 11.

13. Improving the media content editing architecture includes improving the large-scale language model using the contextual information and reward function. The computing device according to claim 11.

14. The reward function is based on one or more audience engagement metrics associated with the published edited media content, wherein the one or more audience engagement metrics include one or more of the following metrics: views, likes, shares, or comments. The computing device according to claim 13.

15. The aforementioned context information includes one or more of the following: conversation history, editing context, or editing draft history. The computing device according to claim 11.

16. A computing system for improving media content editing architecture, A social media network application with a dialog-assisted editing interface, Memory that stores one or more large language models, Equipped with a processor, The processor executes a program using a portion of the memory, A backend tool service including a prompt pool supporting multiple editing tools and multiple application programming interfaces, the dialog-assisted editing interface, and one or more large-scale language models are used to edit media content. Using the aforementioned social media network application, the edited media content is published, The context information related to the editing of the media content is stored in the memory. The system is configured to improve the media content editing architecture using the stored context information. Computing system.

17. The contextual information includes conversation history categorized into user-accepted and user-rejected interactions, and improving the media content editing architecture includes improving the prompt pool based on the user-accepted and user-rejected interactions. The computing system according to claim 16.

18. Improving the media content editing architecture includes improving the one or more large-scale language models using the context information and reward function. The computing system according to claim 16.

19. The reward function is based on one or more audience engagement metrics associated with the published edited media content, wherein the one or more audience engagement metrics include one or more of the following metrics: views, likes, shares, or comments. The computing system according to claim 18.

20. The computer device provides an instruction that causes the computer device to implement the method of claim 1 when executed by the computer device, A non-temporary, computer-readable medium for improving media content editing architecture.

Images