Methods, systems, non-temporary computer-readable storage media, and computer programs for automatically generating effects.

Abstract

This disclosure describes a technique for automatically generating effects. Multiple effect ideas may be generated by at least one trained machine learning model in response to receiving text input from a user. The effect ideas may be decomposed into multiple components depending on the selection of effect ideas. The effect idea may be one of several effect ideas. Executable code may be generated based on the decomposition of the effect ideas. The code may be executed by a predetermined effect creation tool to generate the effect. The generated effect may be output.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This disclosure claims priority to U.S. Utility Patent Application No. 18 / 233,262, filed with the USPTO on August 11, 2023, which is hereby incorporated by reference in its entirety.

[0002] This application relates to a method for automatically generating effects.

Background Art

[0003] Communications carried out using Internet - based tools are increasing. Internet - based tools can be any software or platform. Users may create content and create functions via such Internet - based tools. Improved techniques for content creation and function design via such tools are desirable.

Brief Description of the Drawings

[0004] The following detailed description can be better understood when read in conjunction with the accompanying drawings. For purposes of illustration, exemplary embodiments of various aspects of the present disclosure are shown in the accompanying drawings, but the invention is not limited to the specific methods and means disclosed. [Figure 1] A diagram showing an exemplary system for automatic effect generation according to the present disclosure. [Figure 2] A diagram showing an exemplary user interface for automatic effect generation according to the present disclosure. [Figure 3] A diagram showing an exemplary user interface for automatic effect generation according to the present disclosure. [Figure 4] A diagram showing an exemplary user interface for automatic effect generation according to the present disclosure. [Figure 5] An exemplary flowchart showing script generation according to the present disclosure. [Figure 6] This diagram illustrates an exemplary process for automatic effect generation. [Figure 7] This diagram shows another exemplary process for automatic effect generation. [Figure 8] This diagram shows another exemplary process for automatic effect generation. [Figure 9] This diagram shows another exemplary process for automatic effect generation. [Figure 10] This diagram shows another exemplary process for automatic effect generation. [Figure 11] This diagram shows another exemplary process for automatic effect generation. [Figure 12] This figure shows an exemplary computing device that can be used to perform any of the methods disclosed herein. [Modes for carrying out the invention]

[0005] Communication can take place using internet-based tools that enable users to create content (e.g., images and / or video content) and distribute it to other users for consumption. Such internet-based tools may provide users with various effects (e.g., filters) to use when creating content. Effects are features that can be used to enhance or modify content. Therefore, effects can give content creators more power to express their creativity. Effects may include, for example, beauty filters and / or augmented reality (AR) filters. Beauty filters may be set to enhance facial features, smooth skin, add makeup, or alter facial characteristics. AR filters may be set to overlay digital elements on top of a real-world environment depicted in the content. AR filters may be used to add objects, masks, backgrounds, interactive elements, etc., to the content.

[0006] Effects may be created using effect creation tools (such as EffectHouse or Lens Studio). When creating effects using these tools, the user (e.g., the effect creator) may need to perform complex user interface (UI) operations. However, effect creators may need a considerable amount of domain knowledge or experience to be able to perform such complex UI operations. Therefore, the complexity of such UI operations can create a high barrier to entry for effect creation. Furthermore, effect creation can be excessively time-consuming. For most beginner to intermediate-level effect creators, building an effect that realizes their idea can take an enormous amount of time. Therefore, improved techniques for effect generation are desirable.

[0007] This document describes an improved technique for generating effects. This technique utilizes artificial intelligence (AI) to automatically generate effects when given text input from a user (e.g., an effect creator). Thus, this technique can be used by effect creators with little or no domain knowledge or experience, and may be more efficient (e.g., consume less time) than existing effect generation techniques.

[0008] Figure 1 shows an exemplary system 100 for automatically generating effects, such as effects that can be used to enhance or modify content (e.g., image content, video content, etc.). System 100 may comprise a prompting system 104, an effect creation tool 106, and an effect iteration 108. The prompting system 104, the effect creation tool 106, and the effect iteration 108 may communicate with each other via one or more networks.

[0009] The prompt system 104 may receive user input 102. User input 102 may include any text input. User input 102 may include voice (e.g., acoustic) input. If user input 102 includes voice input, the voice input may be converted to text input using appropriate speech-to-text conversion technology. User input 102 may be received via a user interface. User input 102 received via a user interface may be sent to the prompt system 104 (e.g., forwarded).

[0010] For example, a user may enter one or more characters, symbols, words, phrases, or statements into one or more text boxes on the user interface. For example, a user may enter the word "birthday" into a text box. Depending on what the user has entered into the text boxes, the user may click one or more "Execute" buttons. The characters, symbols, words, phrases, or statements entered by the user may indicate the effect the user wants to generate. For example, if the user enters "birthday" into a text box, this may indicate that the user wants to generate a birthday-themed effect. Based on (for example, accordingly) the user has selected an execute button on the user interface, the characters, symbols, words, phrases, or statements entered by the user may be sent (e.g., forwarded) to the prompt system 104.

[0011] The prompting system 104 may receive user input (e.g., characters, symbols, words, phrases, clauses). The prompting system 104 may generate executable code and / or scripts based on (e.g., using) the user input 102. The prompting system 104 may comprise one or more machine learning models, e.g., large-scale language models. Each of the machine learning models may comprise an artificial intelligence (AI) algorithm. The AI ​​algorithm may utilize deep learning techniques and large-scale datasets to process and understand language (e.g., text). Each of the machine learning models may be trained on a large amount of data and learn language patterns so that it can perform tasks. For example, each of the machine learning models in the prompting system 104 may be trained to process and understand language (e.g., text, user input 102) in order to generate executable code and / or scripts. The machine learning model may be configured to receive user input 102 as input. The machine learning model may be configured to generate executable code and / or scripts based on user input 102. The machine learning model may output the generated executable code and / or scripts.

[0012] In an embodiment, to generate executable code and / or scripts, the machine learning model may generate one or more effect ideas corresponding to user input 102. For example, if user input 102 contains the word "birthday," the machine learning model may generate the effect idea "cake smash." The effect idea "cake smash" may include an AR filter that allows the user to virtually "smash" a cake onto their face (or another area of ​​the image or video content item). The effect idea "cake smash" may also include an AR filter that displays confetti and balloons flying over at least one area of ​​the image or video content item. In addition to the effect idea "cake smash," the machine learning model may generate other (e.g., additional or alternative) effect ideas corresponding to the user input "birthday." If the machine learning model generates two or more effect ideas corresponding to user input 102, the user may select the effect idea they prefer most. For example, a list of effect ideas corresponding to user input may be output (e.g., displayed) to the user interface. The user may select the desired effect idea from the list of effect ideas corresponding to user input 102.

[0013] In one embodiment, to generate executable code and / or scripts, a machine learning model may decompose one of the effect ideas into an effect description (e.g., a component). The effect ideas decomposed by the machine learning model may include an effect idea selected by the user. Alternatively, the effect ideas decomposed by the machine learning model may include an effect idea selected by the machine learning model. For example, the machine learning model may automatically select an effect idea, e.g., a first effect idea, from among several effect ideas. The effect description (e.g., a component) may include one or more of a scene description, an asset description, or an interaction description.

[0014] For example, the effect idea "Cake Smash" may be broken down into an effect description (e.g., components). The effect description associated with the effect idea "Cake Smash" may include a scene description. The scene description may include details indicating that the scene associated with the effect idea includes three-dimensional objects that scatter around, such as a cake, confetti, and balloons. The scene description may also include details indicating that face tracking is used to track the user's face when smashing the virtual cake.

[0015] The effect description associated with the effect idea "Cake Smash" may include an asset description. The asset description may include images of the cake, confetti, and balloons. The asset description may include 3D meshes of the cake and confetti. The asset description may include an image sequence of the balloons scattering. The asset description may include a material that makes the cake look realistic. The asset description may include a visual effect (VFX) shader that adds particles to the scene.

[0016] The effect description associated with the effect idea "Cake Smash" may include an interaction description. The interaction description may include details indicating that the user may interact with the effect by virtually smashing the cake against their face. The interaction description may include details indicating that the user may interact with the effect through screen touch interactions. The interaction description may include details indicating that the scattering of confetti and balloons may be based on a timer or triggered by a smash action.

[0017] In an embodiment, the prompting system 104 (e.g., a machine learning model configured to process language and perform language-related tasks) may determine the components associated with the scene description. The prompting system 104 may determine a detailed list of components associated with the scene description. The detailed list of components may include one or more 3D objects, a face tracker, a screen image, etc. For example, the detailed list of components associated with the scene description of the effect idea "Cake Smash" may include 3D objects, such as a cake, confetti, balloons, etc. The 3D objects may be placed in the 3D space of the effect "Cake Smash". The detailed list of components associated with the scene description of the effect idea "Cake Smash" may include a face tracker configured to track the user's face when "smashing" a virtual cake. The detailed list of components associated with the scene description of the effect idea "Cake Smash" may include a screen image. The screen image may be an image depicting balloons flying. The screen image may be placed on the screen while the effect is in use.

[0018] In an embodiment, the prompting system 104 may determine the assets associated with the asset description. The prompting system 104 may determine a detailed list of assets associated with the asset description. The detailed list of assets may include one or more images, animation keyframes, materials, textures, 3D meshes, diffuse maps, normal maps and / or specular maps, particle systems, image sequences, shaders, etc. One or more machine learning models, for example, artificial intelligence-generated content models, may be used to generate the assets in the asset list. The artificial intelligence-generated content models may include machine learning models trained to assist or replace manual content generation by generating content based on keywords or requests entered by the user.

[0019] For example, the list of assets associated with the asset description for the effect idea "Cake Smash" may include 3D meshes of a cake and confetti. The list of assets associated with the asset description for the effect idea "Cake Smash" may include a diffuse map, a normal map, and / or a specular map configured to make the cake look realistic. The list of assets associated with the asset description for the effect idea "Cake Smash" may include a particle system for making the confetti scattering effect look realistic. The list of assets associated with the asset description for the effect idea "Cake Smash" may include one or more textures (e.g., makeup texture, smearing texture, dripping texture, etc.) for the face tracker component to realistically simulate the cake effect on the user's face. The list of assets associated with the asset description for the effect idea "Cake Smash" may include an image sequence of balloons flying around for the screen image component to add dynamic elements to the scene. The list of assets associated with the asset description for the effect idea "Cake Smash" may include a VFX shader for the screen image component to add particles to make the balloons look more lively and enjoyable.

[0020] In an embodiment, the prompt system 104 may determine an interaction associated with an interaction description. The prompt system 104 may determine a detailed list of interactions associated with the interaction description. The detailed list of interactions may include one or more user interactions associated with the interaction description (e.g., how a user interacts with a component). For example, the detailed list of interactions associated with the interaction description of the effect idea "cake smash" may include a screen tap interaction. The user may tap on the screen to delay or show / hide object interactions for the 3D cake object.

[0021] In an embodiment, the prompt system 104 may generate a component category. The component category may include a scene component, a post - processing component, a face effect component, etc. The prompt system 104 may generate a component category by classifying a detailed list of components into various categories. The prompt system 104 may generate component parameters associated with each component of the component category. The component parameters may include component transformations associated with the scene component. The component parameters may include post - processing parameter filling associated with the post - processing component. The component parameters may include face effect parameter filling associated with the face effect component. For example, the component parameters for the effect idea "cake smash" may include parameters set to place the cake in the center of the user's face. The prompt system 104 may generate executable code and / or scripts based on the component parameters associated with each component of the component category.

[0022] In one embodiment, the prompt system 104 may send executable code and / or scripts to the effect creation tool 106. The effect creation tool 106 may receive executable code and / or scripts. The effect creation tool 106 may execute the code and / or scripts in an execution environment. To execute the code and / or scripts, the effect creation tool 106 may define (e.g., design, create, generate) a set of application programming interfaces (APIs) within the effect creation tool 106 for generating effects from scripts. The effect creation tool 106 may call the APIs to assemble effects corresponding to effect ideas. The effect creation tool 106 may build an execution environment for executing scripts. The effect creation tool 106 may output generated effects by executing the generated scripts. The generated effects may include filters used when creating content. The generated effects may be used to enhance or modify content. The generated effects may include, for example, beauty filters and / or AR filters.

[0023] In an embodiment, effect iteration 108 may allow the user to iterate over the generated effect. Effect iteration 108 may allow the user to input more text to iterate over the generated effect. For example, the user may want to add a new element to the generated effect. As another example, the user may want to delete one or more elements in the generated effect. As yet another example, the user may want to replace one or more elements in the generated effect. Effect iteration 108 may generate a new element using one or more artificial intelligence-generated content models. Effect iteration 108 may add the new element to the generated effect. Effect iteration 108 may delete one or more elements from the generated effect. Effect iteration 108 may replace one or more elements in the generated effect with a new element (e.g., an element generated by an artificial intelligence-generated content model).

[0024] In embodiments, the generated effects may be output. The generated effects may include the final effect (for example, after all iterations have been performed by effect iteration 108, if iterations exist). The output generated effects may be used to modify content, such as image content or video content. For example, the output generated effects may be used to modify content before it is delivered or provided to subscribers of the content service by the content service. The content may include short videos. Short videos may have a duration less than or equal to a predetermined time limit, such as one minute, five minutes, or other predetermined minutes. Not limited to, but as an example, a short video may include at least one and four or fewer 15-second segments combined with each other. Short video durations can provide viewers with quick bursts of entertainment, allowing them to watch a large amount of video within a short time frame. Such quick bursts of entertainment can be popular on social media platforms.

[0025] Figure 2 shows an exemplary UI 200. UI 200 may be configured to accept user text input. UI 200 may include one or more text boxes. Text boxes may include prompt text boxes. The user may enter user input through UI 200. The user may enter user input into the prompt text box. User input may include any text input. User input may include one or more characters, symbols, words, phrases, or statements in the prompt text box. The user may then click one or more "Execute" buttons. The characters, symbols, words, phrases, or statements entered by the user may represent the effect the user wants to generate. As shown in Figure 2, the user has entered the user input "archi!" into the prompt text box. The user may then select the "Execute" button. Depending on the user's selection of the "Execute" button, the user input may be sent (e.g., forwarded) to a prompt system (e.g., prompt system 104). The prompt system 104 may use user input to generate executable code and / or scripts based on (for example, using) the user input. The executable code and / or scripts may be sent to an effect creation tool (for example, effect creation tool 106) for generating effects.

[0026] Figure 3 shows an exemplary UI 300. UI 300 may be configured to display a generated effect. UI 300 may display effect 302. Effect 302 may have been generated using executable code and / or scripts generated by a prompting system based on user input (e.g., by effect creation tool 106). For example, effect 302 may have been generated using executable code and / or scripts generated by prompting system 104 based on user input "archi!" (e.g., by effect creation tool 106). Effect 302 may include an augmented reality (AR) filter. Effect 302 may be configured to overlay digital elements (e.g., the arch character / cartoon shown in Figure 3) onto a real-world environment represented by image or video content.

[0027] Figure 4 shows an exemplary UI 400. UI 400 may be used by the user to perform effect iterations. UI 400 may include iteration settings 402. The user may modify effect 302 by adjusting iteration settings 402. For example, the user may adjust iteration settings 402 to adjust one or more of the position, rotation, scale, texture, stretch mode, blend mode, color, horizontal flip, and vertical flip associated with effect 302. The generated effect may be output. The generated effect may include the final effect (for example, after all iterations have been performed, if iterations exist). The output generated effect may be used to modify content, such as image content or video content.

[0028] Figure 5 is an exemplary flowchart 500. The flowchart illustrates a process for generating executable code and / or scripts. The process may be performed by one or more trained machine learning models to generate executable code and / or scripts based on user input. User input may be received in 502. User input may include arbitrary text input (e.g., one or more characters, symbols, words, phrases, or clauses). User input may include voice input (e.g., sound). If user input includes voice input, the voice input may be converted to text input using appropriate speech-to-text techniques. User input may be received via a user interface.

[0029] In 504, conceptualization output may be performed. A list of effect ideas may be output. The effect ideas may correspond to user input. For example, each effect idea may correspond to an effect associated with user input. In 506, an effect idea may be selected. The effect idea may be selected from a list of effect ideas. The effect idea may be selected by the user. For example, the user may select the effect idea they prefer most. Alternatively, the effect idea may be selected automatically (for example, without user input) by a prompt system (for example, prompt system 104).

[0030] In 508, the effect idea may be broken down. The effect idea may be broken down into effect descriptions (e.g., components). An effect description (e.g., a component) may include one or more of the scene description 512, asset description 514, or interaction description 510. In 516, a detailed list of interactions may be determined. The detailed list of interactions may include one or more user interactions associated with an interaction description (e.g., how a user interacts with a component). In 518, a detailed list of components associated with a scene description may be determined. The detailed list of components associated with a scene description may include one or more 3D objects, face trackers, screen images, etc. In 520, a detailed list of assets associated with an asset description may be determined. The detailed list of assets may include one or more images, animation keyframes, materials, textures, 3D meshes, diffuse maps, normal maps and / or specular maps, particle systems, image sequences, shaders, etc.

[0031] The detailed list of components may be categorized into various categories. In 522, some of the components in the detailed list of components may be classified as scene components. In 524, some of the components in the detailed list of components may be classified as post-processing components. In 526, some of the components in the detailed list of components may be classified as face effect components. Component parameters may be generated for each component in the component category. In 528, component transformations may be generated. Component transformations may be associated with scene components. In 530, post-processing parameter filling may be generated. Post-processing parameter filling may be associated with post-processing components. In 532, face effect parameter filling may be generated. Face effect parameter filling may be associated with face effect components. In 534, executable code and / or scripts may be generated. Executable code and / or scripts may be generated based on the component parameters associated with each component in the component category. In 536, executable code and / or scripts may be sent to an effect creation tool, such as Effect House.

[0032] Figure 6 shows an exemplary process 600 performed by one or more components shown in Figure 1. For example, process 600 may be performed at least partially by system 100. Process 600 may be performed to automatically generate effects. Although shown as a series of operations in Figure 6, those skilled in the art will understand that in various embodiments, the shown operations may be added, deleted, rearranged, or modified.

[0033] In 602, multiple effect ideas may be generated. Multiple effect ideas may be generated by at least one large language model. Multiple effect ideas may be generated in response to receiving text input from a user. User input may include any text input (e.g., one or more characters, symbols, words, phrases, or clauses). User input may be received via a user interface. Each of the multiple effect ideas may correspond to a text input. An effect idea may be selected from among multiple effect ideas. An effect idea may be selected, for example, by a user. Alternatively, an effect idea may be selected automatically (e.g., without user input).

[0034] In 604, the effect idea may be decomposed. The effect idea may be decomposed into multiple components. The effect idea may be decomposed, for example, depending on whether the effect idea is selected by the user. The effect idea may be one of several effect ideas. The multiple components may include one or more of the following: 3D objects, face trackers, screen images, etc. In 606, executable code may be generated. The executable code may be generated based on the decomposition of the effect idea.

[0035] In 608, code may be executed. The code may be executed by a designated effect creation tool. The code may be executed to generate an effect. The effect creation tool may execute code and / or scripts in an execution environment. To execute code and / or scripts, the effect creation tool may define (e.g., design, create, generate) a set of application programming interfaces (APIs) within the effect creation tool for generating effects from scripts. The effect creation tool may call APIs to assemble an effect corresponding to an effect idea. The effect creation tool may build an execution environment for executing scripts. The effect creation tool may output a generated effect by executing the generated script.

[0036] In 610, the generated effects may be output. The output generated effects may be used to modify content, such as image content or video content. For example, the output generated effects may be used to modify content before it is delivered or provided to subscribers of the content service by the content service. The content may include short videos. Short videos may have a duration less than or equal to a predetermined time limit, such as one minute, five minutes, or other predetermined minutes. Not limited to, but as an example, short videos may include at least one, and no more than four, 15-second segments combined with each other. Short video durations can provide viewers with quick bursts of entertainment, allowing them to watch a large amount of video within a short time frame. Such quick bursts of entertainment may become popular on social media platforms.

[0037] Figure 7 shows an exemplary process 700 performed by one or more components shown in Figure 1. For example, process 700 may be performed at least partially by system 100. Process 700 may be performed to automatically generate effects. Although shown as a series of operations in Figure 7, those skilled in the art will understand that in various embodiments, the shown operations may be added, deleted, rearranged, or modified.

[0038] In 702, the effect idea may be broken down. The effect idea may be broken down into multiple components. The effect idea may be broken down, for example, depending on whether the effect idea is selected by the user. The effect idea may be one of several effect ideas. The multiple components may include one or more of the following: 3D objects, face trackers, screen images, etc.

[0039] In 704, an effect scene may be defined. The effect may correspond to an effect idea. The scene may contain multiple components. The scene may include face tracking. In 706, multiple components may be determined. A detailed list of components associated with the scene may be determined. The list of components may include one or more 3D objects, a face tracker, a screen image, etc. For example, the list of components associated with the scene description of the effect idea "Cake Smash" may include 3D objects, such as a cake, confetti, balloons, etc. The 3D objects may be placed in the 3D space of the effect "Cake Smash". The list of components associated with the scene description for the effect idea "Cake Smash" may include a face tracker configured to track a human face when "smashing" a virtual cake. The list of components associated with the scene description for the effect idea "Cake Smash" may include a screen image. The screen image may be an image depicting balloons flying. The screen image may be placed on the screen while the effect is in use.

[0040] In 708, parameters may be generated. Parameters may be associated with multiple components. Multiple components may be classified into different component categories, for example, scene components, post-processing components, and face effect components. Component parameters may be generated that are associated with each component in a component category. Component parameters may include component transformations associated with scene components. Component parameters may include post-processing parameter filling associated with post-processing components. Component parameters may include face effect parameter filling associated with face effect components. For example, component parameters for the effect idea "Cake Smash" may include parameters set to place the cake in the center of the face. Executable code and / or scripts may be generated based on the component parameters. The effect creation tool may generate the effect by executing the code and / or scripts in the execution environment. The output generated effect may be used to modify content, for example, image content or video content.

[0041] Figure 8 shows an exemplary process 800 performed by one or more components shown in Figure 1. For example, process 800 may be performed at least partially by system 100. Process 800 may be performed to automatically generate effects. Although shown as a series of operations in Figure 8, those skilled in the art will understand that in various embodiments, the shown operations may be added, deleted, rearranged, or modified.

[0042] In 802, the effect idea may be broken down. The effect idea may be broken down into multiple components. The effect idea may be broken down depending on the effect idea selected by the user. The effect idea may be one of several effect ideas. The multiple components may include one or more of the following: 3D objects, face trackers, screen images, etc. In 804, multiple assets may be determined. The multiple assets may be for an effect. The effect may correspond to an effect idea. A detailed list of assets associated with the effect may be determined. The list of assets may include one or more images, animation keyframes, materials, textures, 3D meshes, diffuse maps, normal maps and / or specular maps, particle systems, image sequences, shaders, etc. In 806, multiple assets may be generated. One or more artificial intelligence-generated content models may be used to generate the assets.

[0043] Figure 9 shows an exemplary process 900 performed by one or more components shown in Figure 1. For example, process 900 may be performed at least partially by system 100. Process 900 may be performed to automatically generate effects. Although shown as a series of operations in Figure 9, those skilled in the art will understand that in various embodiments, the shown operations may be added, deleted, rearranged, or modified.

[0044] In 902, the effect idea may be broken down. The effect idea may be broken down into multiple components. The effect idea may be broken down depending on the effect idea selected by the user. The effect idea may be one of several effect ideas. The multiple components may include one or more of the following: 3D objects, face trackers, screen images, etc. In 606, executable code may be generated.

[0045] In 904, at least one interaction may be determined. At least one interaction may be determined for an effect. The effect may correspond to an effect idea. At least one interaction may include one or more user interactions (e.g., how the user interacts with multiple components). For example, one or more interactions may include showing or hiding an object in response to receiving user input, e.g., a screen tap gesture, a finger slide gesture, a voice command, a body movement, etc. For example, the user may perform one or more gestures to delay, show and / or hide one or more of the multiple components. In 906, an interaction may be configured. The interaction may be configured to be triggered by a given user gesture. For example, the interaction may be configured to be triggered by a screen tap gesture, a finger slide gesture, a voice command, a body movement, etc.

[0046] Figure 10 shows an exemplary process 1000 performed by one or more components shown in Figure 1. For example, process 1000 may be performed at least partially by system 100. Process 1000 may be performed to automatically generate effects. Although shown as a series of operations in Figure 10, those skilled in the art will understand that in various embodiments, the shown operations may be added, deleted, rearranged, or modified.

[0047] In 1002, a predetermined effect creation tool may be configured. The predetermined effect creation tool may be configured to include a set of application programming interfaces (APIs). The set of application programming interfaces (APIs) may be created within the effect creation tool 106 for generating effects from scripts. The effect creation tool may call APIs to assemble effects corresponding to effect ideas. In 1004, an execution environment may be constructed. The predetermined effect creation tool may receive executable code and / or scripts associated with the effects. The execution environment is constructed to execute the code / scripts.

[0048] In 1006, an effect may be generated. The effect may be generated by calling a set of APIs in the execution environment to execute code and / or scripts. A given effect creation tool may output a generated effect by executing code / scripts generated by a prompting system (e.g., prompting system 104). The generated effect may be used to modify content, such as image content or video content. For example, the generated effect may be used to modify content before it is delivered or provided to subscribers of a content service by a content service. The content may include short videos. Short videos may have a duration less than or equal to a predetermined time limit, such as one minute, five minutes, or other predetermined minutes. As an example, but not limited to, a short video may include at least one and four 15-second segments combined with each other. Short video durations can provide viewers with quick bursts of entertainment, allowing them to watch a large amount of video within a short time frame. Such quick bursts of entertainment may become popular on social media platforms.

[0049] Figure 11 shows an exemplary process 1100 performed by one or more components shown in Figure 1. For example, process 1100 may be performed at least partially by system 100. Process 1100 may be performed to automatically generate effects. Although shown as a series of operations in Figure 11, those skilled in the art will understand that in various embodiments, the shown operations may be added, deleted, rearranged, or modified.

[0050] In 1102, executable code / script may be generated. The executable code / script may be generated based on the decomposition of an effect idea by a prompting system (e.g., prompting system 106). The effect idea may be decomposed into multiple components. The effect idea may be decomposed depending on the selection of an effect idea by the user. The effect idea may be one of several effect ideas generated by the prompting system. The multiple components may include one or more of the following: 3D objects, face trackers, screen images, etc.

[0051] In 1104, code / scripts may be executed. Code / scripts may be executed by a designated effect creation tool. Code / scripts may be executed to generate effects. The effect creation tool may execute code and / or scripts in an execution environment. To execute code and / or scripts, the effect creation tool may define (e.g., design, create, generate) a set of application programming interfaces (APIs) within the effect creation tool for generating effects from scripts. The effect creation tool may call APIs to assemble effects corresponding to effect ideas. The effect creation tool may build an execution environment for executing code / scripts. The effect creation tool may output generated effects by executing the generated code / scripts. In 1106, generated effects may be output. Generated effects may include scenes. Generated effects may include multiple assets. Generated effects may include at least one interaction.

[0052] In 1108, the generated effect may be iterated over. The generated effect may be iterated over based on user input. Iterating over the generated effect may include deleting one or more elements from the generated effect. Iterating over the generated effect may include adding one or more new elements to the generated effect. Iterating over the generated effect may include replacing at least one element in the generated effect with at least one new element. Iterating over the generated effect may include generating new elements using one or more artificial intelligence-generated content models and adding the new elements to the generated effect, deleting one or more elements from the generated effect, and / or replacing one or more elements in the generated effect. A final effect may be output. The final effect is the effect after all iterations have been performed, if iterations exist. The output final effect may be used to modify content, such as image content or video content.

[0053] Figure 12 shows a computing device that can be used in various forms, such as the services, networks, modules, and / or devices shown in Figure 1. With respect to the exemplary architecture of Figure 1, any of the system components may be implemented by one or more instances of the computing device 1200 in Figure 12. The computer architecture shown in Figure 12 may be used to perform any form of the computer described herein, such as a conventional server computer, workstation, desktop computer, laptop computer, tablet, network device, PDA, electronic reader, digital mobile phone, or other computing node, including implementing the methods described herein.

[0054] The computing device 1200 may include a printed circuit board or "motherboard" that can be connected to multiple components or devices via a system bus or other telecommunication path. One or more central processing units (CPUs) 1204 may operate in conjunction with a chipset 1206. The CPUs 1204 may be standard programmable processors that perform arithmetic and logical operations necessary for the operation of the computing device 1200.

[0055] The CPU 1204 may perform the necessary operations by transitioning from one discrete physical state to the next by manipulating switching elements that distinguish and change these states. Switching elements may typically include electronic circuits that maintain one of two binary states, such as flip-flops, and electronic circuits that provide output states based on logical combinations of the states of one or more other switching elements, such as logic gates. These basic switching elements may be combined to create more complex logic circuits, including registers, adders and subtractors, arithmetic logic units, floating-point units, and so on.

[0056] The CPU 1204 may be extended with or replaced by other processing units, such as the GPU 1205. The GPU 1205 may include processing units that are specialized for, but not limited to, highly parallel computing such as graphics and other visualization-related processing.

[0057] Chipset 1206 may provide an interface between the CPU 1204 and the remaining components and devices on the board. Chipset 1206 may also provide an interface to random-access memory (RAM) 1208 used as main memory within the computing device 1200. Chipset 1206 may also provide an interface to a computer-readable storage medium, such as read-only memory (ROM) 1220 or non-volatile RAM (NVRAM) (not shown), for storing basic routines that can start the computing device 1200 and facilitate the transmission of information between various components and devices. According to the embodiments described herein, the ROM 1220 or NVRAM may store other software components necessary for the operation of the computing device 1200.

[0058] The computing device 1200 may operate in a network environment using logical connections to remote computing nodes and computer systems via a local area network (LAN). The chipset 1206 may include functionality for providing network connectivity via a network interface controller (NIC) 1222, such as a Gigabit Ethernet adapter. The NIC 1222 may enable the computing device 1200 to connect to other computing nodes via the network 1216. It should be understood that multiple NICs 1222 may be present within the computing device 1200, enabling the computing device to connect to other types of networks and remote computer systems.

[0059] The computing device 1200 may be connected to a mass storage device 1228 that provides non-volatile storage for the computer. The mass storage device 1228 may store system programs, application programs, other program modules, and data as described in more detail here. The mass storage device 1228 may be connected to the computing device 1200 via a storage controller 1224 connected to a chipset 1206. The mass storage device 1228 may consist of one or more physical storage units. The mass storage device 1228 may include a management component 1212. The storage controller 1224 may interact with the physical storage units via a serial attached SCSI (SAS) interface, a serial advanced technology attachment (SATA) interface, a fiber channel (FC) interface, or other types of interfaces for physically connecting and transmitting data between the computer and the physical storage units.

[0060] The computing device 1200 may store data on the mass storage device 1228 by converting the physical state of the physical storage unit to reflect the stored information. The specific conversion of the physical state may depend on various factors and different embodiments of this specification. Examples of such factors include, but are not limited to, the technology for realizing the physical storage device and whether the mass storage device 1228 is characterized as a primary storage device or a secondary storage device, etc.

[0061] For example, the computing device 1200 may store information in the mass storage device 1228 by issuing commands via the storage controller 1224 to change the magnetic properties of a specific location in a magnetic disk drive unit, the reflective or refractive properties of a specific location in an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of physical media are possible without departing from the scope and spirit of this specification, and the above examples are provided merely to facilitate the explanation. The computing device 1200 may further read information from the mass storage device 1228 by detecting the physical state or characteristics of one or more specific locations in the physical storage unit.

[0062] In addition to the mass storage device 1228 described above, the computing device 1200 may have access to other computer-readable storage media to store and retrieve information such as program modules, data structures, or other data. Those skilled in the art will understand that the computer-readable storage media can be any available medium that provides storage for non-temporary data and is accessible by the computing device 1200.

[0063] Computer-readable storage media may include, but are not limited to, volatile and non-volatile, temporary and non-temporary computer-readable storage media, and removable and non-removable media implemented by any method or technique. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory or other solid-state memory technologies, compact disc ROM (CD-ROM), digital versatile disk (DVD), high-definition DVD (HD-DVD), Blu-ray or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, other magnetic storage devices, or any other media that can be used to store desired information in a non-temporary manner.

[0064] A mass storage device such as the mass storage device 1228 shown in Figure 12 may store an operating system for controlling the operation of the computing device 1200. The operating system may include one version of the LINUX operating system. The operating system may include one version of Microsoft's WINDOWS SERVER operating system. In another embodiment, the operating system may include one version of the UNIX® operating system. Various mobile phone operating systems such as IOS and ANDROID® may also be used. It should be understood that other operating systems may also be used. The mass storage device 1228 may store other systems or applications and data used by the computing device 1200.

[0065] The mass storage device 1228 or other computer-readable storage medium may further encode computer-executable instructions that, when loaded into the computing device 1200, transform the computing device from a general-purpose computing system into a dedicated computer capable of implementing the embodiments described herein. As described above, these computer-executable instructions transform the computing device 1200 by specifying how the CPU 1204 transitions between states. The computing device 1200 may have access to a computer-readable storage medium that stores computer-executable instructions that, when executed by the computing device 1200, can perform the methods described herein.

[0066] A computing device such as the computing device 1200 shown in Figure 12 may further include an input / output controller 1232 for receiving and processing input from multiple input devices such as a keyboard, mouse, touchpad, touchscreen, electronic stylus pen, or other types of input devices. Similarly, the input / output controller 1232 may provide output to a display such as a computer monitor, flat panel display, digital projector, printer, plotter, or other types of output device. It should be understood that the computing device 1200 may not include all the components shown in Figure 12, may include other components not explicitly shown in Figure 12, or may utilize an architecture entirely different from the architecture shown in Figure 12.

[0067] As described herein, the computing device may be a physical computing device such as the computing device 1200 in Figure 12. The computing node may also include a virtual machine host process and one or more virtual machine instances. Computer executable instructions may be executed indirectly by the physical hardware of the computing device by interpretation and / or execution of instructions stored and executed within the context of the virtual machine.

[0068] It should be understood that the methods and systems are not limited to any particular method, component, or embodiment. It should also be understood that the terms used herein are for the purpose of describing a particular embodiment and are not intended to limit it.

[0069] When used in the specification and appended claims, the singular forms “one,” “one,” and “the” include multiple references unless the context explicitly indicates otherwise. Ranges may here be expressed as “about” one particular value and / or “about” another particular value. Where such ranges are expressed, another embodiment includes the range from the one particular value and / or the other particular value. Similarly, when values ​​are expressed as approximations by using the antecedent “about,” it should be understood that the particular value forms other embodiments. Furthermore, it should be understood that each endpoint of a range is significant both with respect to other endpoints and independently of other endpoints.

[0070] "Optional" or "optionally" means that the events or circumstances described later may or may not occur, and the specification includes both cases in which such events or circumstances occur and cases in which they do not occur.

[0071] Throughout this description and claims, the word “includes” and its variations, such as “includes” and “inclusive,” mean “includes, but not limited to,” and are not intended to exclude, for example, other components, integers, or steps. “Exemplary” means “an example of,” and is not intended to indicate a preferred or desirable embodiment. “Like” is used for interpretive purposes, not restrictively.

[0072] The components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groups, etc., of these components, specific references to each of the various individual and collective combinations and permutations of these components may not be explicitly stated, and it should be understood that each of them is specifically assumed and described herein for all methods and systems. This applies to all aspects of this application, including but not limited to the operations in the described methods. Therefore, if there are various additional operations that can be performed, it should be understood that each of these additional operations can be performed in any particular embodiment or combination of embodiments of the described methods.

[0073] The method and system can be more readily understood by referring to the following detailed description of preferred embodiments and examples therein, as well as the accompanying drawings and their descriptions.

[0074] As those skilled in the art will understand, the method and system may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the method and system may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied on the storage medium. More specifically, the method and system may take the form of web-implemented computer software. Any suitable computer-readable storage medium may be used, including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

[0075] Embodiments of the methods and systems are described below with reference to block diagrams and flowcharts of the methods, systems, apparatus, and computer program products. It should be understood that each block in the block diagrams and flowcharts, and each combination of blocks in the block diagrams and flowcharts, may be implemented by computer program instructions. These computer program instructions may be loaded into a general-purpose computer, a dedicated computer, or another programmable data processing device to generate a machine, such that instructions executed on a computer or other programmable data processing device generate means for implementing one or more functions specified in the flowchart.

[0076] These computer program instructions may be stored in computer-readable memory, which can instruct a computer or other programmable data processing device to operate in a particular way so that the instructions stored in computer-readable memory produce a product containing computer-readable instructions for realizing a function defined in one or more blocks of a flowchart. The computer program instructions may also be loaded into a computer or other programmable data processing device so that instructions executed on the computer or other programmable data processing device provide steps for realizing a function defined in one or more blocks of a flowchart, causing the computer or other programmable data processing device to execute a series of operational steps to generate a computer realization process.

[0077] The various characteristics and processes described above may be used independently of each other or combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of this disclosure. Furthermore, in some implementations, some method or process blocks may be omitted. The methods and processes described herein are not limited to any particular order, and the associated blocks or states may be executed in any other appropriate order. For example, the described blocks or states may be executed in an order other than the one specifically described, or multiple blocks or states may be combined within a single block or state. The exemplary blocks or states may be executed sequentially, in parallel, or in any other way. Blocks or states may be added to or removed from the described exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the described exemplary embodiments.

[0078] Furthermore, various items are shown to be stored in memory or on storage devices during use, and it should be understood that these items or parts thereof may be transferred between memory and other storage devices for the purposes of memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or the system may be run in memory on another device and communicate with the illustrated computing system via intercomputer communication. Moreover, in some embodiments, some or all of the system and / or modules may be implemented or provided in other ways, for example, at least in part, by firmware and / or hardware, and the hardware includes, but is not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Modules, systems, and data structures, in whole or in part, may be stored (e.g., as software instructions or structured data) on computer-readable media such as hard disks, memory, networks, or portable media products for reading by appropriate devices or via appropriate connections. Systems, modules, and data structures may also be transmitted (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless and wired / cable media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may take other forms. Therefore, the present invention can be implemented in other computer system configurations.

[0079] While methods and systems have been described in relation to preferred embodiments and specific examples, the embodiments herein are intended to be illustrative rather than restrictive in all embodiments, and the scope is not intended to be limited to specific embodiments.

[0080] Unless otherwise specified, the methods described herein do not require that the operations be performed in a particular order. Therefore, if a method claim does not actually describe the order in which the operations should be performed, or if the claim or specification does not specifically state that the operations are limited to a particular order, it is not intended to infer any order in any manner. This applies to any possible non-expressive grounds for interpretation, including logical issues relating to the arrangement of steps or the flow of operations, simple meanings derived from grammatical structure or punctuation, and the number or type of embodiments described in the specification.

[0081] As will be apparent to those skilled in the art, various modifications and changes are possible without departing from the scope or spirit of this disclosure. Given the specifications and practices described herein, other embodiments will be obvious to those skilled in the art. These specifications and illustrative drawings are intended to be considered illustrative only, and their true scope and spirit are indicated by the following claims.

Claims

1. A method for automatically generating effects, In response to receiving text input, generate multiple effect ideas using at least one trained machine learning model, The process involves decomposing one of the aforementioned multiple effect ideas into multiple components, depending on which effect idea was selected. The process involves generating executable code based on the breakdown of the aforementioned effect idea, The aforementioned code is executed using a predetermined effect creation tool to generate an effect, Outputting the generated effects, A method that includes this.

2. Deconstructing an effect idea is Defining the scene for the aforementioned effect, Determining the aforementioned multiple components, To generate parameters associated with the aforementioned multiple components, The method according to claim 1, including the method described in claim 1.

3. Deconstructing an effect idea further includes determining multiple assets for the said effect. The method according to claim 1.

4. Further including generating the aforementioned multiple assets using at least one other machine learning model, The aforementioned at least one other machine learning model is trained to generate content based on user input. The method according to claim 3.

5. Deconstructing an effect idea further includes determining at least one interaction with the said effect. The method according to claim 1.

6. This further includes configuring the at least one interaction to be triggered by a predetermined user gesture. The method according to claim 5.

7. The generated effect includes a scene, multiple assets, and at least one interaction. The method according to claim 1.

8. Executing the aforementioned code with a predetermined effect creation tool to generate the aforementioned effect is, Configuring the predetermined effect creation tool to include a set of application programming interfaces (APIs), Setting up the execution environment, The execution environment further includes generating the effect by calling the set of APIs and executing the code. The method according to claim 1.

9. The further includes iterating over the generated effect based on user input, Iterating over the generated effect includes removing one or more elements from the generated effect, adding one or more new elements to the generated effect, or replacing at least one element within the generated effect with at least one new element. The method according to claim 1.

10. It is a system, At least one processor, When executed by at least one of the aforementioned processors, the system In response to receiving text input, generate multiple effect ideas using at least one trained machine learning model, The process involves decomposing one of the aforementioned multiple effect ideas into multiple components, depending on which effect idea was selected. The process involves generating executable code based on the breakdown of the aforementioned effect idea, The aforementioned code is executed using a predetermined effect creation tool to generate an effect, Outputting the generated effects, At least one memory having computer-readable instructions for performing an operation including, A system that includes this.

11. Deconstructing an effect idea is Defining the scene for the aforementioned effect, Determining the aforementioned multiple components, To generate parameters associated with the aforementioned multiple components, The system according to claim 10, including the following:

12. Deconstructing an effect idea further includes determining multiple assets for the said effect, The operation further includes generating the plurality of assets using at least one other machine learning model, the at least one other machine learning model being trained to generate content based on user input. The system according to claim 10.

13. Deconstructing an effect idea further includes determining at least one interaction with the said effect, The operation further includes configuring the at least one interaction to be triggered by a predetermined user gesture. The system according to claim 10.

14. The generated effect includes a scene, multiple assets, and at least one interaction. The system according to claim 10.

15. Executing the aforementioned code with a predetermined effect creation tool to generate the aforementioned effect is, Configuring the predetermined effect creation tool to include a set of application programming interfaces (APIs), Setting up the execution environment, The execution environment further includes generating the effect by calling the set of APIs and executing the code. The system according to claim 10.

16. The further includes iterating over the generated effect based on user input, Iterating over the generated effect includes removing one or more elements from the generated effect, adding one or more new elements to the generated effect, or replacing at least one element within the generated effect with at least one new element. The system according to claim 10.

17. A non-temporary computer-readable storage medium storing computer-readable instructions, wherein, when executed by a processor, the computer-readable instructions cause the processor to perform an operation, and the operation is In response to receiving text input, generate multiple effect ideas using at least one trained machine learning model, The process involves decomposing one of the aforementioned multiple effect ideas into multiple components, depending on which effect idea was selected. The process involves generating executable code based on the breakdown of the aforementioned effect idea, The aforementioned code is executed using a predetermined effect creation tool to generate an effect, This includes outputting the generated effects. A non-temporary computer-readable storage medium.

18. Deconstructing an effect idea is Defining the scene for the aforementioned effect, Determining the aforementioned multiple components, To generate parameters associated with the aforementioned multiple components, A non-temporary computer-readable storage medium according to claim 17, including the following:

19. Deconstructing an effect idea further includes determining at least one interaction with the effect, and the operation is The further includes configuring the at least one interaction to be triggered by a predetermined user gesture. A non-temporary computer-readable storage medium according to claim 17.

20. Executing the aforementioned code with a predetermined effect creation tool to generate the aforementioned effect is, Configuring the predetermined effect creation tool to include a set of application programming interfaces (APIs), Setting up the execution environment, The execution environment further includes generating the effect by calling the set of APIs and executing the code. A non-temporary computer-readable storage medium according to claim 17.

Images