System evolved architecture for optimizing media content editing system

Abstract

Examples related to a system evolution architecture for optimizing a media content editing system are provided. One aspect includes a method of optimizing a media content editing architecture, the method comprising: editing media content using a large language model and a back-end tool service, the back-end tool service comprising a cue pool and a plurality of application programming interfaces corresponding to a plurality of editing tools; publishing the edited media content; storing contextual information related to editing of the media content; and optimizing the media content editing architecture using the stored contextual information.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Application Serial No. 18 / 346,737, filed July 3, 2023, entitled "System Evolution Architecture for Optimizing a ('refine') Media Content Editing System", the disclosure of which is incorporated herein by reference in its entirety. BACKGROUND OF THE INVENTION

[0003] Original media content in its original recorded form is typically edited before publication to enhance its appeal for better audience engagement. Editing media content (e.g., images, audio, video, and other forms) typically involves using software with editing capabilities provided in the form of editing tools. Editing of media content can include a variety of operations and modifications. For example, in the case of video editing, editing can include trimming clips, reordering clips, adjusting playback speed, embedding content such as special effects and subtitle text, adjusting audio, cropping, etc. Additionally, the use of powerful editing software enables a non-linear editing (NLE) system in which multiple edits are performed on the original media content in a non-destructive process, such that the original data can be restored, i.e., the edits can be reversed. SUMMARY OF THE INVENTION

[0004] Examples related to a system evolution architecture for optimizing a media content editing system are provided. One aspect includes a method of optimizing a media content editing architecture that includes: using a large language model and a backend tool service to edit media content, the backend tool service including a pool of prompts and a plurality of application programming interfaces corresponding to a plurality of editing tools; publishing the edited media content; storing context information related to the editing of the media content; and using the stored context information to optimize the media content editing architecture.

[0005] This Summary is provided to introduce some concepts in a simplified form that will be further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additionally, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A block diagram model is shown depicting a general pipeline and various components describing an example technical architecture for implementing a media content editing application.

[0007] Figure 2 is a block diagram model illustrating an example backend tool service for providing editing tools and editing capabilities, which can be used in Figure 1is implemented in the general pipeline described in

[0008] Figure 3 is a block diagram model of an example use of the context memory 302 in the illustrated media content editing architecture, which can be implemented in the Figure 1 general pipeline described in

[0009] Figure 4 is a block diagram model of an example system evolution and optimization application for a media content editing architecture, which can be implemented in the Figure 1 general pipeline described in

[0010] Figure 5 is a block diagram model of an example media content editing model architecture with a system evolution and optimization process, which provides a Figure 1 detailed illustration of the general pipeline described in

[0011] Figure 6 is a flowchart of an example method of a media content editing process using machine learning techniques, which can be implemented using the Figure 1 technical architecture of

[0012] Figure 7 is a flowchart of an example method for optimizing a media content editing architecture, which can be implemented using the Figure 1 technical architecture of

[0013] Figure 8 Schematically shows a non-limiting embodiment of a computing system that can implement one or more of the above methods and processes. Detailed Description

[0014] Media content editing software that can provide powerful editing tools is widely used for commercial and personal purposes. Generally, content editing software involves using a user interface (UI) with various sections, menus, buttons, etc. to navigate and select the desired editing tools. Over time, these technologies have evolved, providing a large number of tools for performing a large number of editing tasks. However, software with more powerful editing capabilities and functionality naturally results in greater complexity. Therefore, for ordinary users, many features remain unknown. Complex UI navigation, lack of understanding of the software capabilities, and difficulty in utilizing the functions all result in underutilization of the editing software. For example, a typical user of editing software may not know or lack the ability to use specific tools or feature parts of the software to perform the edits they desire.

[0015] In view of the above observations, a media content editing architecture that utilizes machine learning techniques is provided. The technical architecture utilizing machine learning techniques can be configured in various ways to provide an intuitive media content editing application. Such applications can be configured to receive an editing request from a user and perform one or more desired edits on the media content provided by the user. In some implementations, the media content to be edited is generated by the application. The editing request can be provided in the form of text, and machine learning techniques and natural language processing can be applied to translate the editing request into one or more edits to be performed. The edits can be executed, and the rendered result can be provided to the user for evaluation. In some implementations, the editing process is performed as a non-linear editing (NLE) process. Such implementations better utilize the editing capabilities of the architecture in a more flexible manner. The user can undo an edit or provide another editing request. The process can continue iteratively until the user is satisfied, at which point the user publishes the edited media content.

[0016] Various machine learning techniques, such as deep learning models, can be applied. In some implementations, the media content editing architecture includes a large language model (LLM) that is used to parse and interpret user input to predict one or more editing actions to be performed. The inference prediction can be performed based on a conversational text interaction with the user by receiving the user's text input and responding with a conversational reply. The media content editing architecture can also include a prompt manager for providing prompts in response to an editing request. The prompts can be retrieved from a prompt database. The prompt manager then fills the user's request into the provided prompts and feeds them to the LLM agent, which utilizes the LLM to perform the inference prediction, resulting in a list of instructions or actions that correspond to the edits to be performed. The LLM agent can also be configured to execute the actions to edit the media content. To perform the edits, the LLM agent utilizes a registered database of available editing tools that the agent can access. The database can be linked to the available editing tools and their associated application programming interfaces (APIs) that the LLM agent can utilize to perform the editing operations.

[0017] In some implementations, the media content editing architecture is configured with a system evolution process for training and optimizing the architecture. For example, the LLM and / or the prompt database can be optimized based on the operation history and the edited media content. The media content editing architecture can be configured to save and store session history and / or context information (e.g., asset descriptions of the edited media content). To prevent dilution of valid samples, the media content editing architecture can be configured to save information on successful submissions (e.g., the edited content finally published by the user). The saved information can be used to optimize the media content editing architecture based on a predefined reward function. For example, various indicators associated with the published edited media content can be used to determine the reward function. Such indicators can include indicators representing the success of the edited media content in terms of viewing engagement. Examples of such indicators include views, comments, likes, shares, etc., associated with the published edited media content.

[0018] Turning now to the drawings, a media content editing architecture utilizing machine learning techniques is illustrated and described in more detail. Figure 1 A block diagram model illustrating a computing system 100 is shown. The block diagram model depicts a general pipeline and various components of an example technical architecture for implementing a media content editing application in a client-server environment. The computing system 100 includes a server system 101 that includes a plurality of server computing devices configured to execute the illustrated modules and services to implement a social media network platform. The server system 101 is configured to communicate with a plurality of client computing devices 103 via a computer network N such as the Internet, each client computing device executing a social network client 102. For example, the computing system 100 can implement a short-form video social media network where users create, publish, share, and engage with short-form videos. In other implementations, the computing system 100 can be implemented as an offline application on a computing device. It should be understood that some of the modules shown on the server system can be implemented on client computing devices, such as backend editing tools. Additionally, the social network client can be a mobile client of a social media network, an effect editing software program executed on a personal computer, or other software.

[0019] The editing process is performed through a dialogue-assisted editing interface 104 that includes a dialogue interface 106. A user 108 provides media content 110 to be edited along with an editing request 112. In some implementations, the media content 110 is generated by a media content editing application at the request of the user. For example, generative machine learning techniques can be utilized to generate the media content. The media content 110 can be in various forms. For example, the media content 110 can be an image, a recording, a video, etc. The media content 110 can be displayed through the dialogue-assisted editing interface 104. Additionally, the edits performed on the media content 110 during the editing process can be displayed to the user 108 through the dialogue-assisted editing interface 104, allowing the user 108 to evaluate their next steps.

[0020] The editing request 112 is provided to a prompt manager module 114. In some implementations, the editing request 112 is provided in the form of a text input. In response, the prompt manager module 114 performs a query on a prompt pool 116 to retrieve prompts 118. In some implementations, the retrieved prompts 118 are selected from the prompt pool 118 based on the editing request. The prompt pool 116 can include a database of predefined prompts, and each predefined prompt can be related to the editing capabilities of the computing system 100. For example, the prompts can include a basic description of a given tool, typical problems associated with the tool, the defined input format for the tool, and / or possible intermediate steps when using the tool. Using the prompt pool 116 provides several advantages. One advantage includes the standardization of inputs. Another advantage includes the flexibility in expanding the set of editing tools. For example, when a new tool is added to the editing capabilities of the computing system, the corresponding prompt can be added to the prompt pool 116.

[0021] The prompt manager module 114 populates the retrieved prompts 118 with the editing request 112 and passes it to the LLM agent 120. In some implementations, the computing system 100 includes a content asset analyzer 121 that is used to process the media content 110 to generate metadata, which can be provided as an input to the LLM agent 120. For example, the content asset analyzer 121 can preprocess video content to extract individual frames, analyze the visual and audio content of the video content, and generate video metadata, which can include a text description of the analyzed visual and audio content, the identified entities, timestamps of key events, and video captions of the video content.

[0022] The LLM agent 120 includes an LLM prediction module 122 that utilizes an LLM 124 to perform inferential predictions on received inputs. The LLM 124 can be implemented as a language model formed by a trained neural network with a large number of parameters. The LLM 124 can be trained as a general model or for a limited range of tasks. For example, a media content editing architecture can be implemented with a single general-trained LLM or multiple LLMs, each trained for a different task. In some implementations, a set of LLMs is provided, each trained for a specific range of tasks, and the LLM agent 120 selects which LLM to use based on the received prompt 118. The prompt 118, along with the use of the user's edit request 112, provides structure and context to the input to the LLM 124. As such, the input is somewhat predictable in structure, enabling the LLM 124 to provide more accurate inferential predictions. The LLM agent 120 can be configured to provide an interactive text session with the user 108, where the LLM 124 is used to generate a conversation response 126 and provide it back to the user 108 via the conversation interface 106. The user 108 can then provide a new text input to advance the session. The session continues until the LLM agent 120 determines to terminate the session, which can be based on the new text input and / or the current number of turns in the session. At session termination, the LLM prediction module 122 uses the LLM 124 to generate an inferential prediction based on the received (multiple) text inputs.

[0023] The LLM agent 120 includes an action planning and execution module 128 that parses the inferential predictions to generate a list of (multiple) edit actions. Possible edit actions can be selected from a tool database 130 that lists the editing tools available to the computing system 100 for editing media content 110. The tool database 130 is provided in a backend service 132 that includes tools 134 and associated APIs 136. Tools for editing media content can include, but are not limited to, tools for adding, removing, and / or modifying various forms of content such as text, images, video, audio, etc. For example, a tool can be implemented to embed an audio recording in video content. In some implementations, a generation process is used to create the added content.

[0024] The action planning and execution module 128 uses the appropriate API 136 to call the tools 134 required to perform the (multiple) edit actions to perform the (multiple) edit action list. The (multiple) edit actions are performed on the media content 110 provided by the user 108, and the edited media content 138 is provided back to the user 108 through the dialogue-assisted editing interface 104, where a rendering of the edited media content 138 is displayed for the user 108 to view to determine their next action. For example, the user 108 can decide to revert the performed edits, provide a new edit request 112 for additional edits, or publish 140 the edited media content 138. Upon publishing 140, a copy of the edited media content 138 can be stored on the content server 142. In Figure 1 the depicted example, the published media content 144 is provided on the social network client 102 for other users 146 of the platform to view.

[0025] In some implementations, the media content editing architecture includes a system evolution process that optimizes its ability to more effectively suggest and / or perform actions / edits. Various types of feedback can be used in the system evolution process. For example, direct user feedback (e.g., the user 108 can provide feedback in the form of a rating system that attributes effectiveness to the prompt words and / or tools used in performing the edits). Another example of feedback includes the use of the session history of successful submissions and / or context information (e.g., the published media content 144). Different reward functions can be used to determine the amount of impact for a given optimization iteration. In Figure 1 the shown example, the computing system 100 includes a platform audience engagement aggregation module 148 that provides information about the audience engagement indicators relative to the published media content 144. Example indicators include the number of views / listens, comments, shares, likes, etc. A higher audience engagement indicator means a more "successful" edited media content. Similarly, more weight can be given to the information used in the optimization process related to the published media content with a higher audience engagement indicator. For example, when a predetermined threshold of the audience engagement indicator (e.g., a predetermined number of video views within a predetermined time frame) is reached, an optimization process can be performed for the published media content that reaches the threshold.

[0026] The optimization process can be performed on various modules in the architecture. In Figure 1In the example, computing system 100 includes a prompt optimization module 150 for optimizing the prompt pool 116. Computing system 100 also includes an LLM fine-tuning module 152 for optimizing the LLM 124. The session history and / or context information of the published media content can be used to optimize the prompt pool 116 and / or the LLM 124. For example, it is not practical to provide every editing option for a given prompt. Thus, a set of available options is typically provided for a given prompt. Optimizing the prompt pool affects the set of options provided to the user. By using a high audience engagement indicator as a representative of the success of the edited media content, the session history and / or context information related to the editing of the media content can be used to optimize the options provided in a given prompt, thus providing more "popular" options. As a more specific example, in response to a user request to embed music in video content, the prompt can initially include options regarding different music genres. The prompt can then be optimized to include options for more popular genres based on the published content that shows a high audience engagement indicator when edited to embed music of the popular genres. In this way, computing system 100 can be continuously updated to better respond to user editing requests.

[0027] Figure 2 is an illustration Figure 1 block diagram of aspects of an example configuration of computing system 100. Figure 2 Illustrates an example backend tool service 132 for providing editing tools and editing capabilities that can be used in computing system 100. Example backend tool service 132 provides backend support and capabilities that the LLM agent can use to perform edits on media content (e.g., the action planning and execution module 128 can use backend tool service 132 to perform edits on media content 110).

[0028] Backend tool service 132 includes a repository of available editing tools / capabilities for the media content editing architecture. In some implementations, the tools are arranged and organized in groups. In additional implementations, the tools are organized in several hierarchies. Such an organization scheme allows for conversational interaction, providing the user with a practical number of options for a given choice. For example, instead of listing every available tool for the user to choose from, groups can be provided first to narrow down and give a better context for the edit the user desires.

[0029] In Figure 2In the depicted example, the tools 134 are organized into groups 202. For example, the music recommendation tool 134A and the filter recommendation tool 134N are shown as being organized under the umbrella of the recommendation 202 group. Other groups and classifications shown include understanding, description, AI generation, search and match, localization, structure parsing, AI modification, and evaluation. Each grouping can include various editing capabilities across various modalities such as images, videos, music, text, and speech. Example capabilities of the understanding grouping can include content embedding and tagging. Example capabilities of the localization grouping can include object detection, event detection, letter recognition, object segmentation, event / scene detection, and beat / chorus / start detection. Example capabilities of the description grouping can include image captioning, video captioning, title generation, and text summarization. Example capabilities of the structure parsing grouping can include slicing (shot boundaries) and highlight detection. Example capabilities of the AI generation grouping can include various generation processes for content creation such as image generation, video generation, music generation, and video script generation. Example capabilities of the AI modification grouping can include trimming, volume adjustment, voice alteration, denoising, super-resolution, cropping, background removal, tone mapping, patching, video-audio synchronization, and curve speed. Example capabilities of the search and match grouping can include material search and material replacement. Example capabilities for the recommendation grouping can include the recommendation and application of various content such as filters, music, titles, narrative voice, animation, special effects, stickers, and text (including different fonts, styles, animation, and positions). Example capabilities of the evaluation grouping can include image quality, video quality, and music quality. It can be readily understood that the backend tool service 132 can include any number of groupings 202 using any classification scheme. Additionally, each grouping 202 can include any number of tools 134 which, in some examples, can be further classified into sub-groupings.

[0030] Each tool 134 includes information that describes the callback API 136, which can be used by the LLM agent to invoke the editing tool to perform an edit on media content. The collection of callback APIs 136 is aggregated within the tool API pool 130, which serves as a repository accessible by the LLM agent. For example, as Figure 1 shown, the action planning and execution module 128 utilizes the tool API pool 130 to execute the list of edit actions it generates to form the edited media content 138.

[0031] For each tool 134, a corresponding prompt 204 is generated. The set of prompts 204 is aggregated in a prompt pool 116, which in some examples can be accessed by a prompt manager during prompt retrieval. Prompts 204 can be formatted in various ways. In some implementations, the prompt 204 for a given tool includes a basic description of the tool, typical problems associated with the tool, the defined input format for the tool, and / or intermediate steps when using the tool. The backend tool service 132 can be implemented dynamically by the ability to add and remove tools 134. As new tools 134 are added, corresponding prompts 204 can be generated and added to the backend tool service 132 and thus to the prompt pool 116.

[0032] Figure 3 is a diagram Figure 1 block diagram of aspects of an example configuration of a computing system 100. Figure 3 Illustrates an example use of a context memory 302 in a media content editing architecture that can be used in a computing system 100. Directed connections are illustrated to show the relationships between components associated with the context memory 302. A direct method for editing media content 110 includes a direct command 303 given by a user via a dialogue-assisted editing interface 104. The direct command 303 describes a specific editing action desired by the user and is formatted such that the media content editing architecture understands the given command without using an LLM or an LLM agent 120. As such, the direct command 303 enables the user to directly invoke an editing tool from the backend tool service 132 to perform an edit on the media content 110. For more complex, unstructured queries, the context memory 302 can be used to store context information to help guide the editing process.

[0033] The context memory 302 can include the storage of various context information that can be used by the media content editing architecture for various purposes. For example, during the editing of media content by an action planning and execution module 128, an edit draft history 304 can be compiled based on a list of non-linear edits and associated editing tools. The edit draft history 304 can include steps and edits for rendering the edited media content to the user via the dialogue-assisted editing interface 304. The context memory 302 also includes an edit context 306 that provides context to the tools and editing capabilities provided by the backend tool service 132.

[0034] Conversational interactions between the user and the LLM can also be stored. In Figure 3In the depicted example, the conversation history 308 is stored in the context memory 302. For example, the conversation can be stored when a conversational input is provided to the LLM agent 120 and when the LLM prediction module 122 uses the LLM to generate a conversation reply 126. The conversation history 308 can be used for various purposes. During the editing process, the conversation history 308 can provide information to the media content editing architecture to determine how many rounds of editing have been performed. In some implementations, the media content editing architecture is configured to suggest publishing the edited content 110 after a certain amount of conversational back-and-forth and / or multi-round editing. Another use includes prompt suggestion based on interactions in previous conversations. For example, previous interactions where the user rejected the suggested edited content can be stored in the conversation history 308, and the media content editing architecture can be configured to be less likely to provide related prompts.

[0035] In some implementations, the conversation history 308 and / or the edit draft history 304 can be used to train or optimize the media content editing architecture, which can include optimizing the prompt pool and / or the LLM. The conversation history 308 and / or the edit draft history 304 can be stored for each user submission, and their content can be used to optimize the prompt pool and / or the LLM. For example, the published edited media content with a high audience engagement indicator can be considered a training sample for the optimization process. The conversation history 308 and / or the edit draft history 304 of the published edited media content can be used to optimize the prompt pool and / or the LLM such that the prompts and conversation replies 126 associated with the published edited media content are more likely to appear in future interactions.

[0036] Figure 4 is an illustration Figure 1 of aspects of an example configuration of the computing system 100. Figure 4 illustrates an example system evolution and optimization application of a media content editing architecture that can be used in the computing system 100. In Figure 4In the depicted example, an example system evolution and optimization process is performed for the backend tool service 132 and the LLM 124. Various processes can be used to optimize the media content editing architecture. In some implementations, reinforcement learning algorithms are implemented, such as reinforcement learning from human feedback (RLHF) and proximal policy optimization (PPO). For example, the information recorded from a successful editing process 402 can be used to optimize the prompts provided by the LLM 124 and the backend tool service 132 to provide more relevant prompts and responses in future interactions with the user. Records of successful editing processes can be provided at various stages of the editing process. Various information can be recorded, such as session history and context information (e.g., descriptions of assets). The prompt / query and response pairs in such information can be used as training samples for the optimization process. The session history can include successful conversions and less successful session results. For example, as Figure 3 described in, the context memory 302 can be implemented to store various information about the editing process, such as the session history 308, the edit draft history 304, the editing context 306, etc.

[0037] A "successful" editing process can be defined in multiple ways. In some implementations, an editing process is considered successful when the edited media content is published. At this time, information related to the editing process, such as the session history 308 and the edit draft history 304, is recorded. In other implementations, each editing process interaction with the user is recorded. However, this can generate a large amount of unnecessary data and have little impact on the effectiveness of the prompts and tool suggestions. In other implementations, an editing process of the published edited media content that reaches a predetermined threshold of audience engagement is considered successful.

[0038] In Figure 4In the example depicted, the information logged for a given successful editing process is organized into user-accepted interactions 404 and user-rejected interactions 406. Such interactions can include the user's responses to the suggested edits and tool options provided by the prompt manager and / or the LLM agent. The model 400 includes an editing experience pool 408 that aggregates the records of successful editing processes 402, including the user-accepted and -rejected interactions 404, 406. The aggregated information within the editing experience pool 408 can be used by the prompt optimization module 150 to optimize the backend tool service 132. More specifically, the editing experience pool 408 can be used to optimize the prompts provided by the backend tool service 132. For example, the prompts can be modified based on the information in the editing experience pool 408 that describes efficient and inefficient prompts, which can be associated with the user-accepted and -rejected interactions 404, 406, respectively. User-accepted interactions 404 can provide context suggesting that the prompts accepted by the user may result in more successful editing processes. Thus, similar prompts can be configured to be suggested more frequently in future interactions. Similarly, the prompts associated with user-rejected interactions 406 can be modified accordingly or configured to be suggested less frequently in additional interactions.

[0039] The aggregated information within the editing experience pool 408 can also be used by the LLM fine-tuning module 152 to optimize the LLM 124. Similar to the prompt optimization module 150, the LLM fine-tuning module 152 can utilize the information in the editing experience pool that describes efficient and inefficient interactions as positive and negative reinforcement data, respectively. In some implementations, a reward function is implemented to determine the degree to which the information in the editing experience pool impacts the optimization process. Various reward models can be implemented. In Figure 4 the example depicted, online performance data is used as the reward model 410 to optimize the LLM 124. The online performance data of the published edited media content can be quantified using various audience engagement metrics and indicators, such as views, likes, shares, comments, etc. A platform audience engagement aggregation module 148, such as the module illustrated and described with reference to Figure 1 can be used to aggregate the relevant audience engagement indicators of the published edited media content from the hosting service of the published edited media content. Such data can be fed into the reward model 410 to determine the weights of the training samples (the information in the editing experience pool 408) during the optimization process. Although Figure 4 the evolution and optimization system is depicted as utilizing online performance data as the reward model for the optimization of the LLM 124, such models can also be used for the optimization of the prompts in the backend tool service 132.

[0040] Figure 5 is illustrated as being applicable toFigure 1 Block diagram of an example media content editing model architecture with a system evolution and optimization process for use with computer system 100. Figure 5 A detailed illustration of the pipeline flow of a conversational non-linear editing process using an example content editing model architecture is provided. The process begins with user 108 interacting with the dialogue-assisted editing interface 104 and providing media content 110 to be edited. The media content 110 can be in any form, including images, audio, video, etc. In some implementations, the media content 110 is generated by the example media content editing model architecture through a generative AI process. The dialogue-assisted editing interface 104 can be implemented on any computing device. In some implementations, the dialogue-assisted editing interface 104 is provided within a social network client, such as Figure 1 within the social network client 102 depicted in. As described above, the social network client can include various social network platforms, such as short-form video social media platforms.

[0041] The dialogue-assisted editing interface 104 provides an interface through which user 108 can view the media content 110 during the editing process, such as rendering the selected edited results. Additionally, the dialogue-assisted editing interface 104 includes a dialogue interface 106 that can send and receive text commands. The editing process includes user 108 providing an editing request 112 using the dialogue interface 106. The editing request 112 is provided to the prompt manager module 114. Since the editing capabilities of the example media content editing model architecture can include many editing tools, the prompt manager module 114 can be implemented to help structure and narrow the editing request to a subset of the architecture's editing capabilities. Through prompt engineering, the prompt manager module 114 and the prompt retrieval module 502 operate to retrieve prompts from the prompt pool 116. The retrieved prompts are typically related to the editing request 112. For example, if the editing request 112 is related to music, the prompt retrieval module 502 can query the prompt pool 116 to retrieve music-related prompts. In some implementations, the query provides a set of prompts with similar descriptions to match the editing request 112, and this prompt set is combined with fixed prompts for the tools related to the editing request 112 to form new prompts.

[0042] Generally, the prompt pool 116 includes at least one prompt corresponding to each editing capability. The prompt pool 116 can be implemented as a dynamic database where prompts can be added, deleted, and modified, thus providing flexibility in the expansion of the architecture's set of editing tools. For example, when a new tool is registered to the tool set, the corresponding prompt can also be added to the prompt pool 116. Prompts can be formatted in various ways. In some implementations, the prompts include a basic description of the tool, typical problems related to the tool, the defined input format of the tool, and / or possible intermediate steps when using the tool.

[0043] The edit request 112 and the retrieved prompt can be input into the LLM prediction module 122 of the LLM agent 120, which uses the LLM 124 to perform inferential prediction. The LLM agent 120 can be implemented as a text command sender / receiver that provides conversational interaction with the user 108, where the LLM agent 120 uses the LLM 124 to predict responses to the text inputs (edit request 112 and prompt) it receives. Since the prompt is generally predefined, the LLM 124 can output structured results. In some implementations, the LLM 124 is a single general-purpose LLM. In other implementations, the LLM agent 120 can access a repository of LLMs, each of which is trained for one or more specific tasks. In such cases, which LLM to use can be based on the edit request 112 and / or the prompt.

[0044] The LLM agent 120 can be configured to transform the structured results of the LLM 24 into a tool execution sequence and the inputs for the tool execution. The LLM agent 120 includes an LLM output parser 504 that parses the prediction response from the LLM prediction module 122 and retrieves the structured information in the prediction response. The LLM agent 120 also includes an LLM action planning module 128A and an LLM tool execution model 128B. The LLM action planning module 128A and the LLM tool execution model 128B can be implemented similarly to Figure 1 the action planning and execution module 128. The LLM action planning module 128A can be implemented to plan the actions to be taken based on the structured information. Based on the planned actions, the LLM tool execution module 128B forms a tool chain and executes the tool chain for the tools in the tool chain using API calls from the tool API pool 130. For unsolved problems or complex requests, the LLM agent 120 can use the LLM 124 to perform self-exploration and generate several intermediate steps using the self-exploration module 506 and the tool execution chain module 508 respectively. For each step, the LLM 124 can use search or follow-up questions to gradually approach the final answer. Conversational back-and-forth text can be implemented. For example, the dialogue reply 126 and subsequent responses can be provided to the user 108 through the dialogue interface 106 of the dialogue-assisted editing interface 104. In some implementations, the dialogue replies are stored in the context memory 302, which records the conversation history 308.

[0045] When executing the toolchain, API calls to the tools in the toolchain are utilized to edit media content 110. The backend tool service 132 provides editing capabilities and executes editing steps, storing the steps in the edit draft history 304 in the context memory 302. The edited media content is provided to the user 108 via the dialogue-assisted editing interface 104, and the user 108 can determine their next action. For example, the user 108 can schedule a resumption of editing, provide additional edit requests, or publish 140 the edited media content.

[0046] The model 500 includes a system evolution architecture that can implement the optimization process of the media content editing architecture. The optimization process can be implemented using similar components and methods as described in the reference. Figure 4 In the Figure 5 depicted example, the prompt optimization module 150 and the LLM fine-tuning module 152 are implemented to optimize the prompts within the backend tool service 132 and the LLM 124 respectively. The training data for the optimization process can include various context information stored during the editing process. For each submission (the set of interactions between the given media content 110 and the user 108), the session history 302, as well as context information such as the description of the asset, can be stored in the context memory 302. The session history can include successful conversions and less successful session results. In some implementations, only successful submissions are saved. A "successful" submission can be defined in multiple ways. For example, when publishing 140 the edited media content, the submission can be considered successful.

[0047] After the edited media content is published, the record of the successful editing process 402 is retrieved. Such records can include the context data stored during the editing process of the edited media content, such as the data stored in the context memory 302. In some examples, the context data can be divided into user-accepted interactions 404 and user-rejected interactions 406. Such interactions can include the user's responses to the proposed edits and tool options. The edit experience pool 408 aggregates the context data, which is then used by the prompt optimization module 150 and the LLM fine-tuning module 152 to optimize the backend tool service 132 and the LLM 124 respectively.

[0048] The reward model 410 can be implemented to assign different weights to the training data. The rewards can be based on various criteria. In the Figure 5In the illustrated example, online performance data in the form of an audience engagement indicator is used as a reward function. A higher audience engagement indicator results in a higher reward for the training data (context data) that generates the published edited content. The audience engagement indicator can include various metrics related to the online performance data of the published edited media content. Example indicators include views, likes, shares, comments, etc. A platform view engagement aggregation module 148, such as the module illustrated and described with reference to Figure 1 the illustrated and described module, can be used to aggregate relevant audience engagement indicators for the published edited media content from the hosting service of the published edited media content. In some implementations, the reward model 410 is similarly applied to the prompt optimization module 150.

[0049] Figure 6 is a flowchart of an example method 600 for an media content editing process using machine learning techniques. Such a method can be performed on an media content editing architecture, such as the media content editing architecture illustrated and described in Figure 5 In step 602, method 600 includes receiving media content from a user. Various types of media content and formats can be utilized. For example, the media content can be an image, a recording, or a video. The media content can be provided by the user, such as through an upload process. In some implementations, the media content is provided by a generative AI process.

[0050] In step 604, method 600 includes receiving an edit request from the user for the media content. The edit request can be received from the user by using a conversation-assisted editing interface. Generally, the edit request is received in the form of a text input. The edit request can be received by using a prompt manager module. The edit request can include a request to revert a previous edit to the media content. In some implementations, the edit request can be a direct command in a structured format that allows the media content editing architecture to directly access the editing tools.

[0051] In step 606, method 600 includes editing the media content based on the edit request to generate edited media content. The editing of the media content can be performed using various processes. Sub-steps 606A to 606C describe one such process. In sub-step 606A, method 600 includes retrieving a prompt from a prompt pool. The prompt can be retrieved by using a prompt manager module. The prompt pool can include multiple prompts, where each prompt corresponds to at least one editing tool.

[0052] At sub-step 606B, method 600 includes using a large language model to parse the retrieved prompt and editing request to generate one or more edit actions to be performed on the media content. An LLM agent can be used to receive the input and feed the input into the large language model. The use of the prompt can allow for more structured input, enabling the large language model to provide more consistent responses. The large language model can be configured to parse the input to generate one or more edit actions in the form of a list of action tools.

[0053] At sub-step 606C, method 600 includes performing one or more edit actions on the media content to generate edited media content. Performing the edit actions can include using API calls to the corresponding editing tools. The APIs can be retrieved from a tool API pool.

[0054] At step 608, method 600 includes optionally publishing the edited media content. The edited media content can be published on various platforms. For example, the edited media content can be published on a video social media network in short content form.

[0055] Figure 7 is a flowchart illustrating an example method 700 for optimizing a media content editing architecture. Optimizing the media content editing architecture can be implemented using a system evolution architecture, such as Figure 4 the architecture illustrated and described in. At step 702, method 700 includes using a media content editing architecture, such as Figure 5 the media content editing architecture illustrated and described in to edit the media content. Figure 6 The method described in can also be used to edit the media content. Various types and forms of media content can be utilized. For example, the media content can be an image, a recording, or a video. The media content editing architecture can include a large language model and a backend tool service. The backend tool service can include a prompt pool.

[0056] At step 704, method 700 includes publishing the edited media content. The edited media content can be published on various platforms. For example, the edited media content can be published on a video social media network in short content form.

[0057] At step 706, method 700 includes storing context information related to media content editing. Examples of context information include conversational history, editing context, and edit draft history. In some implementations, the context information includes an asset description of the edited media content. In some implementations, the context information is stored in a context memory. The context information can be used for various purposes. During the editing process, the context information knows the historical actions of the editing process, which can affect the conversational responses of the media content editing architecture. For example, if the context information includes a conversational history where the user has rejected a given proposed edit, the media content editing architecture can be configured not to suggest the edit for the given editing process. Another use of the context information includes the optimization of the media content editing architecture.

[0058] At step 708, method 700 includes using the stored context information to optimize the media content editing architecture. Optimizing the media content editing architecture can include optimizing the prompt pool and / or the large language model. In some implementations, the stored context information includes a session history, which is classified into user-accepted interactions and user-rejected interactions, and optimizing the media content editing architecture includes optimizing the prompt pool based on the user-accepted interactions and the user-rejected interactions. For example, compared to user-rejected interactions, the prompts in the prompt pool can be optimized to suggest relevant edit actions corresponding to user-accepted interactions. The optimization process can include using an audience engagement indicator associated with the published edited media content as a reward function. Example audience engagement indicators include views, likes, shares, and comments. In some implementations, the optimization process is executed when a predetermined threshold of the audience engagement indicator is reached (e.g., a predetermined number of video views within a predetermined time frame).

[0059] In some embodiments, the methods and processes described herein can be associated with a computing system of one or more computing devices. In particular, such methods and processes can be implemented as a computer application or service, an application programming interface, a library, and / or other computer program products.

[0060] Figure 8 A non-limiting embodiment of a computing system 800 that can implement one or more of the above methods and processes is schematically shown. Computing system 800 is shown in a simplified form. Computing system 800 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices such as smartwatches and head-mounted augmented reality devices.

[0061] A media content editing architecture can be implemented to provide intuitive editing tools and experiences for ordinary users. By combining an LLM with various editing tools, users can perform powerful editing on media content without the need for extensive software knowledge. Such an architecture can receive input as unstructured text from the user and, together with the use of prompt words and natural language processing techniques, predict the desired editing requests and use an available pool of editing tools to execute the predictions. Additional implementations can include the optimization of such a technical architecture. Using online performance data of the published edited content enables the system to self-develop and optimize without the expensive and labor-intensive training process of traditional LLM models.

[0062] Computing system 800 includes a logical processor 802, volatile memory 804, and a non-volatile storage device 806. Computing system 800 can optionally include a display subsystem 808, an input subsystem 810, a communication subsystem 812, and / or Figure 8 other components not shown.

[0063] The logical processor 802 includes one or more physical devices configured to execute instructions. For example, the logical processor can be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions can be implemented to perform tasks, implement data types, transform the state of one or more components, achieve a technical effect, or otherwise achieve a desired result.

[0064] The logical processor can include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor can include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processors of the logical processor 802 can be single-core or multi-core, and the instructions executed thereon can be configured for sequential, parallel, and / or distributed processing. The various components of the logical processor can optionally be distributed among two or more separate devices, which can be located remotely and / or configured for cooperative processing. Aspects of the logical processor can be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In such cases, it should be understood that these virtualized aspects run on different physical logical processors of various different machines.

[0065] The non-volatile storage device 806 includes one or more physical devices configured to store instructions executable by the logical processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 806 can be transformed, for example, to store different data.

[0066] The non-volatile storage device 806 may include removable and / or built-in physical devices. The non-volatile storage device 806 may include optical memories (e.g., CD, DVD, HD-DVD, etc.), semiconductor memories (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or magnetic memories (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage device technologies. The non-volatile storage device 806 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that the non-volatile storage device 806 is configured to save instructions even when the non-volatile storage device 806 is powered off.

[0067] The volatile memory 804 may include a physical device that includes random access memory. The volatile memory 804 is typically used by the logic processor 802 to temporarily store information during software instruction processing. It should be understood that when the volatile memory 804 is powered off, the volatile memory 804 generally does not continue to store instructions.

[0068] Aspects of the logic processor 802, the volatile memory 804, and the non-volatile storage device 806 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGA), application specific programs and application specific integrated circuits (PASIC / ASIC), application specific programs and application specific standard products (PSSP / ASSP), system on a chip (SOC), and complex programmable logic devices (CPLD).

[0069] The terms "module", "program", and "engine" may be used to describe an aspect of the computing system 800, which is typically implemented in software by a processor to perform a specific function using a portion of the volatile memory, the function involving a transformation process that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated using a portion of the volatile memory 804 via the logic processor 802 that executes instructions saved by the non-volatile storage device 806. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may encompass individual or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0070] When the display subsystem 808 is included, the display subsystem 808 can be used to present a visual representation of the data stored by the non-volatile storage device 806. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 808 can also be transformed to visually represent the changes in the underlying data. The display subsystem 808 can include one or more display devices utilizing almost any type of technology. Such display devices can be combined with the logic processor 802, volatile memory 804, and / or non-volatile storage device 806 in a shared enclosure, or such display devices can be peripheral display devices.

[0071] When the input subsystem 810 is included, the input subsystem 810 can include one or more user input devices, such as a keyboard, mouse, touch screen, or game controller, or be connected to them. In some embodiments, the input subsystem can include selected natural user input (NUI) components or interface with the NUI. Such components can be integrated or peripheral, and the conversion and / or processing of input actions can be handled on-board or off-board. Example NUI components can include a microphone for voice and / or sound recognition; infrared, color, stereo, and / or depth cameras for machine visualization and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; and / or any other suitable sensors.

[0072] When the communication subsystem 812 is included, the communication subsystem 812 can be configured to communicatively couple the various computing devices described herein to each other and to communicate with other devices. The communication subsystem 812 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem can allow the computing system 800 to send and / or receive messages to and / or from other devices via a network such as the Internet.

[0073] The following paragraphs provide additional description of the subject matter of the present disclosure. One aspect provides a method for optimizing a media content editing architecture, the method comprising: using a large language model and a backend tool service to edit media content, the backend tool service including a pool of prompts and a plurality of application programming interfaces corresponding to a plurality of editing tools; publishing the edited media content; storing context information related to the editing of the media content; and using the stored context information to optimize the media content editing architecture. In this aspect, additionally or alternatively, optimizing the media content editing architecture includes optimizing the large language model and the pool of prompts. In this aspect, additionally or alternatively, the stored context information includes a session history, the session history being classified into user-accepted interactions and user-rejected interactions, and wherein optimizing the media content editing architecture includes optimizing the pool of prompts based on the user-accepted interactions and the user-rejected interactions. In this aspect, additionally or alternatively, optimizing the media content editing architecture includes using the context information and a reward function to optimize the large language model. In this aspect, additionally or alternatively, the reward function is based on one or more audience engagement indicators associated with the published edited media content. In this aspect, additionally or alternatively, the one or more audience engagement indicators include a metric that is one or more of the following: views, likes, shares, or comments. In this aspect, additionally or alternatively, optimizing the large language model includes using the context information to optimize the large language model when the one or more audience engagement indicators reach a predetermined threshold. In this aspect, additionally or alternatively, the predetermined threshold includes reaching a predetermined number of views within a predetermined amount of time since the edited media content was published. In this aspect, additionally or alternatively, the context information includes one or more of the following: session history, editing context, or editing draft history. In this aspect, additionally or alternatively, the media content is published on a social media platform in a short content format. Additionally, in this aspect, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium including instructions that, when executed by a computing device, cause the computing device to implement the methods described herein.

[0074] On the other hand, there is provided a computing device for optimizing a media content editing architecture, the computing device including: a processor and a memory of the computing device, the processor being configured to execute a program using a portion of the memory for: editing media content using a large language model and a backend tool service, the backend tool service including a prompt pool and a plurality of application programming interfaces corresponding to a plurality of editing tools; publishing the edited media content; storing context information related to the editing of the media content; and using the stored context information to optimize the media content editing architecture. In this aspect, additionally or alternatively, the stored context information includes a session history, the session history being classified into user-accepted interactions and user-rejected interactions, and wherein optimizing the media content editing architecture includes optimizing the prompt pool based on the user-accepted interactions and the user-rejected interactions. In this aspect, additionally or alternatively, optimizing the media content editing architecture includes using the context information and a reward function to optimize the large language model. In this aspect, additionally or alternatively, the reward function is based on one or more audience engagement indicators associated with the published edited media content, and wherein the one or more audience engagement indicators include one or more of the following: metrics, the metrics being views, likes, shares, or comments. In this aspect, additionally or alternatively, the context information includes one or more of the following: session history, editing context, or editing draft history.

[0075] On the other hand, there is provided a computing system for optimizing a media content editing architecture, the computing system including: a social media network application including a conversation-assisted editing interface; a memory storing one or more large language models; a processor configured to execute a program using a portion of the memory for: editing media content using the conversation-assisted editing interface, one or more large language models, and a backend tool service, the backend tool service including a prompt pool and a plurality of application programming interfaces corresponding to a plurality of editing tools; publishing the edited media content using the social media network application; storing context information in the memory, wherein the context information is related to the editing of the media content; and using the stored context information to optimize the media content editing architecture. In this aspect, additionally or alternatively, the context information includes a session history, the session history being classified into user-accepted interactions and user-rejected interactions, and wherein optimizing the media content editing architecture includes optimizing the prompt pool based on the user-accepted interactions and the user-rejected interactions. In this aspect, additionally or alternatively, optimizing the media content editing architecture includes using the context information and a reward function to optimize one or more large language models. In this aspect, additionally or alternatively, the reward function is based on one or more audience engagement indicators associated with the published edited media content, and wherein the one or more audience engagement indicators include one or more of the following: metrics, the metrics being views, likes, shares, or comments.

[0076] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, as many variations are possible. The particular routines or methods described herein may represent one or more of any number of processing strategies. As such, the various acts illustrated and / or described may be performed in the order illustrated and / or described, in other orders, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0077] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of various processes, systems, and configurations, as well as other features, functions, acts, and / or properties disclosed herein, and any and all equivalents thereof.

Claims

1. A method for optimizing a media content editing architecture, the method comprising: Using a large language model and a backend tool service to edit media content, the backend tool service including a prompt pool and a plurality of application programming interfaces corresponding to a plurality of editing tools; Publishing the edited media content; Storing context information related to the editing of the media content; And Using the stored context information to optimize the media content editing architecture.

2. The method according to claim 1, wherein optimizing the media content editing architecture comprises: Optimizing the large language model and the prompt pool.

3. The method according to claim 1, wherein the stored context information includes a session history, the session history being classified into interactions accepted by the user and interactions rejected by the user, and wherein optimizing the media content editing architecture includes: Optimizing the prompt pool based on the interactions accepted by the user and the interactions rejected by the user.

4. The method according to claim 1, wherein optimizing the media content editing architecture comprises: Using the context information and a reward function to optimize the large language model.

5. The method according to claim 4, wherein the reward function is based on one or more audience engagement indicators associated with the published edited media content.

6. The method according to claim 5, wherein the one or more audience engagement indicators include metrics that are one or more of the following: view, like, share, or comment.

7. The method according to claim 5, wherein optimizing the large language model comprises: When the one or more audience engagement indicators reach a predetermined threshold, using the context information to optimize the large language model.

8. The method according to claim 7, wherein the predetermined threshold includes reaching a predetermined number of views within a predetermined amount of time since the edited media content was published.

9. The method according to claim 1, wherein the context information includes one or more of the following: session history, editing context, or editing draft history.

10. The method according to claim 1, wherein the media content is published on a social media platform in a short content format.

11. A computing device for optimizing a media content editing architecture, the computing device comprising: A processor and a memory of the computing device, the processor being configured to use a portion of the memory to execute a program for: Using a large language model and a backend tool service to edit media content, the backend tool service including a prompt pool and a plurality of application programming interfaces corresponding to a plurality of editing tools; Publishing the edited media content; Storing context information related to the editing of the media content; And Using the stored context information to optimize the media content editing architecture.

12. The computing device according to claim 11, wherein the stored context information includes a session history, the session history being classified into interactions accepted by the user and interactions rejected by the user, and wherein optimizing the media content editing architecture includes: Optimizing the prompt pool based on the interactions accepted by the user and the interactions rejected by the user.

13. The computing device according to claim 11, wherein optimizing the media content editing architecture includes: Using the context information and a reward function to optimize the large language model.

14. The computing device according to claim 13, wherein the reward function is based on one or more audience engagement indicators associated with the published edited media content, and wherein the one or more audience engagement indicators include metrics that are one or more of the following: view, like, share, or comment.

15. The computing device according to claim 11, wherein the context information includes one or more of the following: session history, editing context, or editing draft history.

16. A computing system for optimizing a media content editing architecture, the computing system comprising: A social media network application, the social media network application including a conversation-assisted editing interface; A memory that stores one or more large language models; A processor configured to use a portion of the memory to execute a program for: Editing media content using the conversation-assisted editing interface, the one or more large language models, and a backend tool service, the backend tool service including a prompt pool and a plurality of application programming interfaces corresponding to a plurality of editing tools; Publishing the edited media content using the social media network application; Storing context information in the memory, where the context information is related to the editing of the media content; And Optimizing the media content editing architecture using the stored context information.

17. The computing system according to claim 16, wherein the context information includes a session history, the session history being classified into interactions accepted by the user and interactions rejected by the user, and wherein optimizing the media content editing architecture includes: Optimizing the prompt pool based on the interactions accepted by the user and the interactions rejected by the user.

18. The computing system according to claim 16, wherein optimizing the media content editing architecture includes: Optimizing the one or more large language models using the context information and a reward function.

19. The computing system according to claim 18, wherein the reward function is based on one or more audience engagement indicators associated with the published edited media content, and wherein the one or more audience engagement indicators include metrics that are one or more of: views, likes, shares, or comments.

20. A non-transitory computer-readable medium for optimizing a media content editing architecture, the non-transitory computer-readable medium including instructions that, when executed by a computing device, cause the computing device to implement the method of claim 1.

Images