Techniques for model-based element creation in special effect creation tool

Abstract

Embodiments of the present disclosure relate to techniques for model-based element creation in a special effect creation tool. Examples are described for creating an element in a special effect creation tool, including: receiving, via a user interface provided for the special effect creation tool, a natural language string requesting creation of an element in the special effect creation tool; providing an input including at least a portion of the natural language string to the model; based on the input, receiving an output string of an expected syntax corresponding to the created element from the model; mapping the output string to one or more commands for creating a format of an element in a special effect creation tool; and providing the one or more commands to a special effect creation tool to cause creation of the element in the special effect creation tool.

Description

Background Art

[0001] Various aspects described relate to special effects creation tools, and more particularly to creating elements in special effects creation tools.

[0002] There are game engines for simplifying the creation of video games or other video-based applications or features by providing a number of video processing or rendering frameworks and tools that facilitate the creation of video games or other video-based applications or features using the frameworks. The tools may include special effects creation tools, such as game engine-based tools, and may include a user interface or other mechanism that allows a user to specify elements to be created in the special effects creation tool, such as layouts, images, etc. to be included when creating video applications, video special effects, and / or corresponding features. Game engine-based tools can generate corresponding instructions with a grammar that the game engine can process based on interaction with the user interface. The game engine can generate corresponding video-based applications, video special effects, or features accordingly based on the grammar generated by the tool.

[0003] Social media applications can also use game engines for certain functions. For example, video capture and editing applications can use game engines to display captured videos, display images or animations or other video effects on the displayed video to allow interaction with the displayed video, etc. For example, a mask can be created and displayed on the face of the video subject, or an avatar can be created and displayed on the video. Special effects creation tools connected to the game engine interface can be used to create video effects (e.g., masks, avatars, etc.) or certain elements used when creating video effects for use with video capture and editing applications. Using special effects creation tools can require its users to have specific knowledge of how to create different elements or other features (such as resources, components, special effects, game elements, etc.) in special effects creation tools and artistic ability to adapt the created or modified textures to be compatible with the video or other scenes on which the assets are used, etc. Summary of the invention

[0004] A simplified summary of one or more implementations is presented below to provide a basic understanding of such implementations. This summary is not an extensive overview of all contemplated implementations, and is neither intended to identify key or important elements of all implementations nor to describe the scope of any or all implementations. Its sole purpose is to present some concepts of one or more implementations in a simplified form as a prelude to a more detailed description that is presented later.

[0005] In an example, a computer-implemented method for creating an element in a special effect creation tool is provided, comprising: receiving, via a user interface provided for the special effect creation tool, a natural language string requesting creation of an element in the special effect creation tool; providing an input including at least a portion of the natural language string to a model; receiving, based on the input, an output string from the model corresponding to an expected syntax for creating the element; mapping the output string to one or more commands in a format for creating the element in the special effect creation tool; and providing the one or more commands to the special effect creation tool to cause creation of the element in the special effect creation tool.

[0006] In another example, a device for creating an element in a special effect creation tool is provided, comprising one or more processors and one or more non-volatile memories having instructions thereon. When executed by the one or more processors, the instructions cause the one or more processors to: receive a natural language string requesting to create an element in the special effect creation tool via a user interface provided for the special effect creation tool; provide an input including at least a portion of the natural language string to a model; based on the input, receive an output string from the model corresponding to an expected syntax for creating the element; map the output string to one or more commands in a format for creating the element in the special effect creation tool; and provide the one or more commands to the special effect creation tool to cause the creation of the element in the special effect creation tool.

[0007] In another example, one or more non-transitory computer-readable storage media storing instructions are provided, which, when executed by one or more processors, cause the one or more processors to perform a method for creating an element in a special effect creation tool. The method includes: receiving a natural language string requesting to create an element in the special effect creation tool via a user interface provided for the special effect creation tool; providing an input including at least a portion of the natural language string to a model; based on the input, receiving an output string from the model corresponding to an expected syntax for creating the element; mapping the output string to one or more commands in a format for creating the element in the special effect creation tool; and providing the one or more commands to the special effect creation tool to cause the creation of the element in the special effect creation tool.

[0008] To accomplish the foregoing and related ends, one or more implementations include the features fully described below and particularly pointed out in the claims. The following description and accompanying drawings set forth in detail certain illustrative features of one or more implementations. However, these features are indicative of only a few of the various ways in which the principles of various implementations may be employed, and this description is intended to include all such implementations and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1is a schematic diagram of an example of a system for using a model when creating elements in a special effects creation tool according to examples described herein.

[0010] Figure 2 is a flow chart of an example of a method for using a model for creating an element in a special effects creation tool according to examples described herein.

[0011] Figure 3 is a block diagram illustrating an example of interaction with a model according to examples described herein.

[0012] Figure 4 is a schematic diagram of an example of an apparatus for performing the functions described herein. DETAILED DESCRIPTION

[0013] The specific embodiments described below in conjunction with the accompanying drawings are intended to be descriptions of various configurations and are not intended to represent the only configurations in which the concepts described herein can be practiced. The specific embodiments include specific details to provide a thorough understanding of the various concepts. However, it is apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, well-known components are shown in block diagram form to avoid confusing such concepts.

[0014] The present disclosure describes various examples related to using models such as machine learning (ML) models to assist in creating elements in special effects creation tools. Special effects creation tools may include applications that allow the creation of video applications (e.g., games, social media applications, etc.), visual effects or video effects for video applications or other applications, video features for video applications or other applications, augmented reality (AR) or virtual reality (VR) special effects for video applications or other applications, etc. For example, a special effects creation tool may be provided to operate with a game engine to allow the creation of video applications, visual effects, AR / VR special effects, etc. to be rendered using a game engine. As described, a game engine may provide a framework for video processing or rendering, and an application or tool (including a special effects creation tool) may use the framework to create video applications, video effects, etc. using a game engine. In one example, a special effects creation tool may include a game engine-based tool developed in conjunction with a game engine to provide a mechanism for creating videos or video effects using user interface commands. A game engine may provide a platform or framework for rendering video via a display device, wherein the game engine may include low-level instructions for rendering video via one or more processors (e.g., a central processing unit (CPU) and / or a graphics processing unit (GPU)), and may expose instructions or interfaces that other applications (e.g., game engine-based tools) may use to cause the game engine to render certain graphics or video.

[0015] According to various aspects described herein, a model can be used to create video effects in a special effects creation tool based on, for example, natural language requests entered via a user interface (UI) in the special effects creation tool. For example, the special effects creation tool may include a UI that includes options for entering a natural language query to cause the creation of certain elements in the special effects creation tool, such as resources (e.g., textures, materials, etc.), visual effects, game elements (e.g., entities, components, scripts), etc. For example, the UI may include options in certain steps when creating a video application (e.g., a game or social media application or special effects) that allows the generation of elements. A natural language request can be provided to the model to obtain a resulting grammar for creating elements in a video application. A parser can parse the resulting grammar and map it to commands (e.g., application programming interface (API) calls) for providing to the special effects creation tool to create elements.

[0016] In this regard, a user using the special effects creation tool can use natural language requests to create elements in the special effects creation tool without requiring the user to have the knowledge or artistic ability to create the elements. This can result in an improved user experience when using the special effects creation tool, and can allow users who may not be sophisticated in video special effects creation or processing to use natural language requests to create elements for use in video creation, rather than having to know and use other processes in the special effects creation tool to create the elements.

[0017] As used herein, a separate or combined processor, at least one processor and / or one or more processors configured to perform or operable to perform multiple actions means including at least two different processors capable of performing different, overlapping or non-overlapping subsets of multiple actions or a single processor capable of performing all actions in multiple actions. In a non-limiting example of multiple processors capable of performing different actions in multiple actions in combination, the description of a processor, at least one processor and / or one or more processors configured to or operable to perform actions X, Y and Z may include at least a first processor configured to or operable to perform a first subset of X, Y and Z (e.g., perform X) and at least a second processor configured to or operable to perform a second subset of X, Y and Z (e.g., perform Y and Z). Alternatively, the first processor, the second processor and the third processor may be configured to or operable to perform a corresponding action in actions X, Y and Z, respectively. It should be understood that any combination of one or more processors may be configured to or operable to perform any one or any combination of multiple actions, respectively.

[0018] As used herein, a separate or combined memory, at least one memory, and / or one or more memories configured to store or having stored thereon instructions executable by one or more processors to perform a plurality of actions means including at least two different memories capable of storing different, overlapping, or non-overlapping subsets of instructions for performing a plurality of actions or a single memory capable of storing instructions for performing all of the plurality of actions. In a non-limiting example of a separate or combined one or more memories capable of storing different subsets of instructions for performing different actions in a plurality of actions, a description of a memory, at least one memory, and / or one or more memories configured or operable to store or having stored thereon instructions for performing actions X, Y, and Z may include: at least a first memory configured or operable to store or having stored thereon a first subset of instructions for performing a first subset of X, Y, and Z (e.g., instructions for performing X); and at least a second memory configured or operable to store or having stored thereon a second subset of instructions for performing a second subset of X, Y, and Z (e.g., instructions for performing Y and Z). Alternatively, the first memory, the second memory, and the third memory may be configured to store or have stored thereon a corresponding instruction subset of a first instruction subset for performing X, a second instruction subset for performing Y, and a third instruction subset for performing Z. It should be understood that any combination of one or more memories may be configured to store or be operable to store or have stored thereon any one or any combination of instructions executable by one or more processors to perform any one or any combination of a plurality of actions. Moreover, one or more processors may be coupled to at least one of the one or more memories, respectively, and configured to or be operable to execute instructions to perform a plurality of actions. For example, in the above non-limiting example of different instruction subsets for performing actions X, Y, and Z, a first processor may be coupled to a first memory storing instructions for performing action X, and at least a second processor may be coupled to at least a second memory storing instructions for performing actions Y and Z, and the first processor and the second processor may execute the corresponding instruction subsets in combination to complete the execution of actions X, Y, and Z. Alternatively, the three processors may access one of three different memories, each storing one of the instructions for performing X, Y, or Z, and the three processors may execute the respective subsets of instructions in combination to complete the execution of actions X, Y, and Z. Alternatively, a single processor may execute instructions stored on a single memory or distributed across multiple memories to complete the execution of actions X, Y, and Z.

[0019] Now go to Figures 1 to 4, examples are depicted with reference to one or more components and one or more methods that can perform the actions or operations described herein, where the components and / or actions / operations in dashed lines may be optional. Figure 2 The operations described in the description are presented in a particular order and / or performed by example components, but in some examples, the order of actions and components performing the actions may be changed depending on the implementation. Moreover, in some examples, one or more of the actions, functions, and / or components described may be performed by a specially programmed processor, a processor executing specially programmed software or computer-readable media, or any other combination of hardware components and / or software components capable of performing the described actions or functions.

[0020] Figure 1 1 is a schematic diagram of an example of a system for using a model when creating elements in a special effects creation tool according to various aspects described herein. The system includes a device 100 (e.g., a computing device) that includes (multiple) processors 102 (e.g., one or more processors) and / or a memory / multiple memories 104 (e.g., one or more memories). In an example, the device 100 may include (multiple) processors 102 and / or a memory / multiple memories 104 that are configured to execute or store instructions or other parameters related to providing an operating system 106 that can execute one or more applications, services, etc. The one or more applications, services, etc. may include a special effects creation tool 110, which may be or include an application that facilitates the creation of a video, an application that includes a video (e.g., a game), a video special effect, or other video feature, wherein a game engine 122 (e.g., also executed via the operating system 106) may render the video to a display 108. For example, the processor(s) 102 and the memory / memories 104 may be separate components communicatively coupled via a bus (e.g., on a motherboard or other portion of a computing device, on an integrated circuit such as a system on a chip (SoC), etc.), components integrated with each other (e.g., the processor(s) 102 may include the memory / memories 104 as an onboard component 101), etc. In other examples, the processor(s) 102 may include multiple processors 102 of multiple devices 100, the memory / memories 104 may include multiple memories 104 of multiple devices 100, etc. The memory / memories 104 may store instructions, parameters, data structures, etc. for use / execution by the processor(s) 102 to perform the functions described herein.

[0021] Additionally, the device 100 may include substantially any device that may have a processor(s) 102 and a memory / memories 104, such as a computer (e.g., a workstation, a server, a personal computer, etc.), a personal device (e.g., a cellular phone (such as a smart phone), a tablet computer, etc.), a smart device (such as a smart TV), etc. Moreover, in examples, various components or modules of the device 100 may be within a single device as shown, or distributed to different devices that are communicatively coupled to each other (e.g., in a network).

[0022] The special effects creation tool 110 may include a user interface module 112 for generating a user interface for output to the display 108 of the device 100 (or, the display of another device). For example, the user interface module 112 may accept input interactions on the displayed user interface from a user to create a video, an application (e.g., a game) including a video, or other video features (e.g., special effects of a video), etc., as described herein. For example, the interaction may include creating certain elements in a video, an application, etc., such as resources, components, special effects, game elements, etc., for adding to or modifying the video. In addition, the user interface module 112 may output to the display 108 of the device 100, such as outputting a video being created, outputting a menu with interactive options for creating a video, such as elements that can be added to or used to modify an image or video. In addition, in an example, the special effect creation tool 110 may optionally include: a model query module 114 for querying a model for obtaining an expected grammar from a natural language request; a grammar parsing module 116 for parsing or mapping the grammar to a command for creating an element in the special effect creation tool 110; an element creation module 118 for creating an element in the special effect creation tool 110 based on the command mapped from the grammar; and / or a model training module 120 for training a model based on the natural language request to obtain the desired image result of the request. In addition, in some examples, the device 100 may communicate with a model 128 (e.g., an AI or ML model), which may communicate with a remotely located model 128 via a network 124, or the model 128 may be stored in a memory / multiple memories 104.

[0023] In an example, the special effects creation tool 110 may provide one or more UIs (e.g., graphical UIs) via a user interface (UI) module 112 that facilitate the creation of video applications, video features, etc. for execution using a game engine 122. The game engine 122 may provide a platform for rendering video on a display 108 or other display device using one or more processors 102 (e.g., a central processing unit (CPU) and / or a graphics processing unit (GPU)). For example, the special effects creation tool 110 may include a video creation studio application with features such as options for creating a canvas for a video, inserting textures, overlays, etc. into a video, a video preview window for previewing the created video, etc. In various examples, the special effects creation tool 110 may support operations including adding elements for creating or modifying an image or video, and may support operations in one or more UIs provided via the user interface module 112. Various aspects described herein relate to using a model to create elements in the special effects creation tool 110 using natural language requests, which may reduce the complexity of using the special effects creation tool 110.

[0024] In an example, once an option is selected or engaged through interaction from a user via the user interface for one or more operations provided by the user interface module 112, the model query module 114 may query the model 128 with certain inputs to retrieve one or more image results. For example, the user interface module 112 may receive a natural language request for creating an element used in the special effect creation tool 110 from the user interaction. The model query module 114 may provide at least a portion of the natural language request as input to the model 128. In an example, the model query module 114 may also provide information about the expected syntax of the output generated based on the input to the model 128. The expected syntax may include output format information, a set of output values ​​that may be returned, etc. In an example, the model query module 114 may receive an output string from the model 128 in the expected syntax. The syntax parsing module 116 may parse the output string to generate a command for providing to the element creation module 118 for creating the element in the special effect creation tool 110. For example, the element creation module 118 may accordingly cause the element to be created in the special effect creation tool 110 (e.g., as a new asset, component, special effect, game element, etc.), which may include storing associated parameters of the element in the memory / memory 104, loading the element from the memory / memory 104 for display on a UI provided by the user interface module 112, etc. In an example, the user interface module 112 may provide the element as a selectable option for incorporation into a video application or feature created using the special effect creation tool 110.

[0025] In an example, the model training module 120 can provide training data to the model 128 to customize the results received from subsequent queries to the model 128. For example, the training data can include a collection of natural language requests and associated output strings, information including expected grammar (e.g., a code base of the special effect creation tool 110, documentation or tutorials for the special effect creation tool 110, etc.). In another example, the training data can be based on feedback received from a user (e.g., via a UI provided by the user interface module 112) indicating whether the results provided by the model 128 in response to the natural language request accurately represent the intent of the user request.

[0026] Figure 2 1 is a flowchart of an example of a method 200 for using a model for creating an element in a special effect creation tool according to various aspects described herein. For example, the method 200 can be performed by a device 100 executing a special effect creation tool 110 and / or one or more components thereof to create an element based on a natural language request by using a model.

[0027] In method 200, in action 202, a natural language string requesting to create an element in a special effect creation tool may be received via a UI provided by a special effect creation tool. In an example, a user interface module 112, for example, in combination with (multiple) processors 102, memory / multiple memories 104, operating system 106, special effect creation tool 110, etc., may receive a natural language string requesting to create an element in a special effect creation tool via a UI provided by the special effect creation tool. For example, the user interface module 112 may display various UIs allowing a user to interact with the UI via the display 108 to select options for creating video content (such as videos, games, video special effects, etc.) using the special effect creation tool. For example, the UI may include options or operations for creating certain elements for adding or modifying video content. As described, elements may include resources, components, special effects, game elements, etc. As part of the options for creating elements, the user interface module 112 may provide a user with a mechanism such as a text input box to enter a natural language request related to creating one or more elements. For example, on a UI for creating video content including special effects, resources or other assets, components, etc., the user interface module 112 may include displaying a text box to request a selection to create a special effect using natural language. For example, the text box may allow a user to type a natural language request for creating a particular element.

[0028] In method 200, in act 204, an input including at least a portion of a natural language string may be provided to a model. In an example, a model query module 114 (e.g., in conjunction with processor(s) 102, memory / memories 104, operating system 106, special effects creation tool 110, etc.) may provide an input including at least a portion of a natural language string to a model (e.g., model 128). For example, model query module 114 may provide a received natural language string, may add context parameters or strings to the natural language string for use by model 128, may remove items from the natural language string, etc. As described, model 128 may be remotely located or stored on device 100 (e.g., in memory / memories 104). In one example, model query module 114 may query multiple models based on the input.

[0029] In method 200, in action 206, an output string of expected syntax corresponding to the created element can be received from the model based on the input. In an example, the model query module 114 (e.g., in conjunction with (multiple) processors 102, memory / multiple memories 104, operating system 106, special effects creation tool 110, etc.) can receive an output string of expected syntax corresponding to the created element from a model (e.g., model 128) based on the input. For example, the model can be trained or otherwise configured to provide an output of the expected syntax. For example, the expected syntax can include a set of category parameter names (which can correspond to element types and / or parameters of element types), and for each parameter name in the multiple parameter names, multiple possible parameter values ​​(also referred to as candidate parameter values).

[0030] For example, to create a light effect, the expected syntax may include a category parameter "Lights" with multiple possible values ​​"DirectionalLight", "PointLight", "SpotLight", etc. that may be supported by the effects creation tool 110. To create an object, the expected syntax may include a category parameter "Objects" with multiple possible values ​​"CubeMesh", "SphereMesh", "PlaneMesh", etc. that may be supported by the effects creation tool 110. To create an effect, the expected syntax may include a category parameter "Effects" with multiple possible values ​​"FaceMask", "HairSegmentation", etc. that may be supported by the effects creation tool 110. To create a material, the expected syntax may include a category parameter "Materials" with multiple possible values ​​"Unlit", "PBR", etc. that may be supported by the effects creation tool 110. In an example, the expected syntax for a texture may include additional information for applying the texture, such as where to locate the texture, how to apply the texture, etc., such as "LibraryMap[Texture,3Dmodel...]". In this regard, for example, model 128 may be trained or configured to provide output strings in the format of [Category]:[Value], such as "[Objects]:[SphereMesh]".

[0031] In the method 200, optionally in act 208, an indication of the expected grammar of the output string can be provided to the model. In an example, the model query module 114 (e.g., in conjunction with the processor(s) 102, the memory / memories 104, the operating system 106, the special effect creation tool 110, etc.) can provide the model (e.g., the model 128) with an indication of the expected grammar of the output string. In this regard, for example, the model query module 114 can provide the indication as an additional input to the model 128 as an instruction for preparing the output string based on the natural language string input. For example, the model query module 114 can provide the format of the expected grammar, the values ​​that can be used and supported by the grammar parsing module 116 (e.g., the value and possible values ​​of the category parameter), etc. as inputs to the model 128. In an example, the model query module 114 can provide the code base of the special effect creation tool 110, the documentation or tutorial of the special effect creation tool 110, etc. as inputs to the model 128. Based on this information, the model 128 can obtain the natural language string input and can provide an output string of the expected grammar.

[0032] In method 200, in act 210, the output string may be mapped to one or more commands in a format for creating an element in a special effect creation tool. In an example, for example in conjunction with processor(s) 102, memory / memories 104, operating system 106, special effect creation tool 110, etc., the syntax parsing module 116 maps or parses the output string to one or more commands in a format for creating an element in a special effect creation tool. For example, the syntax parsing module 116 may include instructions for mapping certain items received in an expected syntax (e.g., category parameters and corresponding value pairs) to commands that the special effect creation tool 110 can understand (e.g., commands supported by an API of the special effect creation tool 110). In one example, the output string may include multiple commands (e.g., multiple parameter / value pairs), and the syntax parsing module 116 may map each of the multiple commands to an associated command (e.g., an API call) in the special effect creation tool 110.

[0033] In a specific non-limiting example, for the natural language string "add a grid to create a tennis ball", the model 128 may return an output string in the following format:

[0034] Objects:SphereMesh

[0035] Texture:Green fur tennis ball

[0036] Material:Unlit

[0037] The syntax parsing module 116 can parse the output string into commands (e.g., API calls) in the special effect creation tool 110 to create a net, generate a texture from the studio or obtain a texture from a texture library corresponding to a green tennis ball, create a new material, and attach dependencies (e.g., textures). This can enable the special effect creation tool 110 to create an element based on the received command, and the special effect creation tool 110 can create the element, include the element in a user interface for selection and application to the video content being created, etc.

[0038] In the example, some elements may have multiple creation steps and multiple associated parameters. For example, a mask may have an associated texture, an associated skin color, etc., where the natural language string indicates such texture, skin color, etc. In this example, the output string for the mask object may include a mask category parameter, a parameter indicating a texture, a parameter indicating a skin color, etc. In this example, the syntax parsing module 116 may parse the output string into multiple associated commands, such as a command to create a mask object, a command to retrieve a texture from a library or generate a texture in a studio, a command to modify a texture based on a skin color, a command to apply a texture in a mask shape, etc.

[0039] In another example, the natural language string may indicate the creation of a script that can be used with an element of the video content (e.g., an element created by the element creation module 118, etc.) during a game. In this example, the model query module 114 can provide the natural language string to the model 128, and can receive an output string including a list of various elements to be created in the special effects creation tool 110 as an output from the model 128, which elements represent interactions during the game and associated special effects to be rendered on the video content. For example, the natural language string may indicate the execution of a special effect on the video based on an interaction during the game (such as a touch or swipe on a touch screen). In this example, the element creation module 118 can create a script to cause the special effect to be performed during the game based on the interaction.

[0040] In a specific example, this functionality can be combined with special effect creation. For example, when a user slides or taps the touch screen, a natural language string can indicate the creation of a special effect that liquefies the image and increases or decreases the liquefaction intensity. In this example, providing a natural language string to the model 128 can cause the output string of the element to cause the creation of a liquefaction special effect, and cause the script to cause the liquefaction intensity to be modified based on the touch screen interaction. In this example, the element creation module 118 can create special effects used in the special effects creation tool 110, and can associate game scripts with special effects used during the game. For example, the output string from the model 128 can be a syntax supported by the script in the special effects creation tool 110.

[0041] In method 200, in action 212, one or more commands may be provided to a special effects creation tool to cause the creation of an element. In an example, an element creation module 118 (e.g., in conjunction with (multiple) processors 102, memory / multiple memories 104, operating system 106, special effects creation tool 110, etc.) may provide one or more commands to the special effects creation tool 110 to cause the creation of a special effect. In an example, the element creation module 118 may call one or more API functions of the special effects creation tool 110 based on the command to create the element. For example, in the above example, the element creation module 118 may call an API to create a net object in an object list of the video content being created, call an API to retrieve a green wool tennis texture from a studio or library, call an API to create a material based on the texture, and apply the material to the net of the newly created element. In addition, the user interface module 112 may display a UI with the newly created element as a selectable element applied to the video content. In an example, the special effects creation tool 110 may facilitate the creation of video content to store and view later or apply special effects to real-time video content. Thus, for example, element creation module 118 may create elements for use with stored video content being created, or for application to real-time video content (eg, while real-time video content is being captured and / or displayed).

[0042] In method 200, optionally in act 214, training data may be provided to the model. In an example, a model training module 120 (e.g., in conjunction with processor(s) 102, memory / memories 104, operating system 106, special effects creation tool 110, etc.) may provide training data to a model (e.g., model 128). For example, the model training module 120 may provide feedback received from user interaction with a user interface regarding whether the created element(s) are related to a natural language string received via the user interface, the degree or rating of the created element(s) relative to the natural language string, etc. In another example, as described above, the model training module 120 may provide training data including an expected grammar of an output string to the model.

[0043] Figure 3300 is a block diagram illustrating an example of an interaction 300 with a model 128 according to various aspects described herein. For example, the interaction 300 may include an input interaction with the model 128 or an output interaction with the model 128. For example, the user interface module 112 may provide a user prompt 302 to the UI for creating an element for creating video content. In the user prompt 302, the user may enter a natural language string for creating an image. The model query module 114 may provide at least a portion of the natural language string to the model 128 for creating the element in the special effect creation tool 110. Additionally, in an example, the model query module 114 may provide an indication of an expected syntax of the output string to the model 128. The model query module 114 may receive an output string for a natural string input from the model 128, wherein the output string may be an expected syntax, as described. The model query module 114 may provide the output string to a command parser 304 (e.g., a syntax parsing module 116) for parsing the output string into a command for creating an element 306.

[0044] When creating an element, the command may be mapped to a configuration for creating the element or modifying the characteristics of the element, etc. The configuration may include a mask configuration 308 for creating a mask object, a sphere configuration 310 for creating a sphere object, a spotlight configuration 312 for creating a spotlight object, etc. The configuration may correspond to creating different elements and may have additional associated characteristics that may be specified by the output string (e.g., depending on the output string format provided to the model 128 or used to train the model 128). For example, the mask configuration 308 may also accept a texture parameter that provides a texture (e.g., an image) of the mask, a skin color parameter that may be used to shade the mask, etc. Thus, for example, the output string may include a mask category parameter and / or one or more characteristic parameters, such as texture, skin color, etc., where the natural language string includes or implies the values ​​of the associated parameters.

[0045] Figure 4 The diagram shows a device 100 ( Figure 1 ) Examples of similar or identical devices 400 include Figure 1 Additional optional component details are shown. In one implementation, device 400 may include processor(s) 402, which may be similar to processor(s) 102, for performing processing functions associated with one or more components and functions described herein. Processor(s) 402 may include a single or multiple collections of processors or multi-core processors. Moreover, processor(s) 402 may be implemented as an integrated processing system and / or a distributed processing system.

[0046] The device 400 may also include a memory / memories 404, which may be similar to the memory / memories 104, such as for storing local versions of applications executed by the processor(s) 402, such as the special effects creation tool 110, related modules, instructions, parameters, etc. The memory / memories 404 may include the types of memory that can be used by a computer, such as random access memory (RAM), read-only memory (ROM), tape, magnetic disk, optical disk, volatile memory, non-volatile memory, and any combination thereof.

[0047] Further, the device 400 may include a communication module 406 that provides for establishing and maintaining communications with one or more other devices, participants, entities, etc. utilizing the hardware, software, and services described herein. The communication module 406 may carry communications between modules on the device 400 and between the device 400 and external devices, such as devices located on a communication network and / or devices serially or locally connected to the device 400. For example, the communication module 406 may include one or more buses and may also include a transmit chain module and a receive chain module associated with a wireless or wired transmitter and receiver, respectively, operable to interface with external devices.

[0048] Additionally, the device 400 may include a data repository 408, which may be any suitable combination of hardware and / or software, that provides mass storage for information, databases, and programs employed in conjunction with the implementations described herein. For example, the data repository 408 may be or may include a data repository for applications and / or related parameters (e.g., special effects creation tool 110, related modules, instructions, parameters, etc.) executed by (multiple) processors 402 or not currently executed by (multiple) processors 402. In addition, the data repository 408 may be a data repository for the special effects creation tool 110, related modules, instructions, parameters, etc., and / or one or more other modules of the device 400.

[0049] Device 400 may include a user interface module 410 that is operable to receive input from a user of device 400 and may also be operable to generate output for presentation to the user. User interface module 410 may include one or more input devices, including but not limited to a keyboard, a numeric keypad, a mouse, a touch-sensitive display, navigation keys, function keys, a microphone, a voice recognition component, a gesture recognition component, a depth sensor, a gaze tracking sensor, a switch / button, any other mechanism capable of receiving input from a user, or any combination thereof. Further, user interface module 410 may include one or more output devices, including but not limited to a display interface, a speaker, a tactile feedback mechanism, a printer, any other mechanism capable of presenting output to a user, or any combination thereof. User interface module 410 may include user interface module 112 or may communicate with it to allow input via user interface module 112 or receive output via user interface module 112 for display, etc.

[0050] By way of example, an element or any part of an element or any combination of elements may be implemented with a "processing system" comprising one or more processors. Examples of processors include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform various functionalities described throughout this disclosure. One or more processors in a processing system may execute software. Software should be broadly interpreted as representing instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, programs, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description languages, or other.

[0051] Therefore, in one or more implementations, the one or more functions described can be implemented with hardware, software, firmware or any combination thereof. If implemented in software, the function can be stored on a computer-readable medium or encoded as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Disks and optical disks used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs) and floppy disks, wherein disks usually reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media.

[0052] The previous description is provided so that any person skilled in the art can practice the various implementations described herein. For those skilled in the art, various modifications of these implementations are easily apparent, and the general principles defined herein can be applied to other implementations. Therefore, the claims are not intended to be limited to the implementations shown herein, but to the full scope consistent with the language claims, wherein unless specifically so provided, references to singular elements are not intended to represent "one and only one", but "one or more". Unless otherwise specifically provided, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various implementations described herein known or later known to those of ordinary skill in the art are intended to be covered by the claims. Moreover, whether or not such disclosure is explicitly stated in the claims, the content disclosed herein is not intended to be dedicated to the public. Unless the element is explicitly stated using the phrase "parts for...", any claim element should not be interpreted as a part plus function.

Claims

1. A computer-implemented method for creating an element in a special effects creation tool, comprising: receiving, via a user interface provided for the special effect creation tool, a natural language string requesting creation of an element in the special effect creation tool; providing an input to the model comprising at least a portion of the natural language string; receiving, from the model, an output string corresponding to an expected syntax for creating the element based on the input; mapping the output string to one or more commands in a format for creating the element in the special effect creation tool; as well as The one or more commands are provided to the special effect creation tool to create the element in the effect creation tool.

2. The computer-implemented method of claim 1 , further comprising: An indication of the expected syntax of the output string is provided to the model.

3. The computer-implemented method of claim 2, wherein the indication of the expected syntax comprises a plurality of parameter names and, for each parameter name in the plurality of parameter names, a plurality of candidate parameter values, wherein the output string comprises at least one parameter name in the plurality of parameter names and, for each parameter name in the at least one parameter name in the plurality of parameter names, at least one candidate parameter value in the plurality of candidate parameter values. 4 . The computer-implemented method of claim 3 , wherein the plurality of parameter names correspond to elements that can be created in the special effect creation tool.

5. The computer-implemented method of claim 1 , wherein providing the one or more commands to the special effects creation tool comprises: An application programming interface API function provided by the special effect creation tool is called to create the element.

6. The computer-implemented method of claim 1, wherein the special effects creation tool creates the element as a selectable element for application to stored or real-time video content.

7. The computer-implemented method of claim 1, wherein the element comprises one or more of the following in the special effect creation tool: a texture, a material, a visual effect, or a game element.

8. An apparatus for creating an element in a special effects creation tool, the apparatus comprising one or more processors and one or more non-volatile memories having instructions thereon, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: receiving, via a user interface provided for the special effect creation tool, a natural language string requesting creation of an element in the special effect creation tool; providing an input to the model comprising at least a portion of the natural language string; receiving, from the model, an output string corresponding to an expected syntax for creating the element based on the input; mapping the output string to one or more commands in a format for creating the element in the special effect creation tool; as well as The one or more commands are provided to the special effect creation tool to create the element in the special effect creation tool.

9. The apparatus of claim 8, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: provide an indication of the expected grammar of the output string to the model.

10. An apparatus according to claim 9, wherein the indication of the expected syntax includes a plurality of parameter names and, for each parameter name in the plurality of parameter names, a plurality of candidate parameter values, wherein the output string includes at least one parameter name in the plurality of parameter names and, for each parameter name in the at least one parameter name in the plurality of parameter names, at least one candidate parameter value in the plurality of candidate parameter values. 11 . The apparatus according to claim 10 , wherein the plurality of parameter names correspond to elements that can be created in the special effect creation tool.

12. The apparatus of claim 8, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: provide the one or more commands to the special effect creation tool at least in part by calling an application programming interface (API) function provided by the special effect creation tool to create the element.

13. The apparatus of claim 8, wherein the special effects creation tool creates the element as a selectable element for application to stored or real-time video content.

14. The apparatus of claim 8, wherein the element comprises one or more of the following in the special effect creation tool: a texture, a material, a visual effect, or a game element.

15. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for creating an element in a special effect creation tool, wherein the method comprises: receiving, via a user interface provided for the special effect creation tool, a natural language string requesting creation of an element in the special effect creation tool; providing an input to the model comprising at least a portion of the natural language string; receiving, from the model, an output string corresponding to an expected syntax for creating the element based on the input; mapping the output string to one or more commands in a format for creating the element in the special effect creation tool; as well as The one or more commands are provided to the special effect creation tool to create the element in the special effect creation tool.

16. The one or more non-transitory computer-readable storage media of claim 15, the method further comprising: An indication of the expected syntax of the output string is provided to the model.

17. One or more non-transitory computer-readable storage media according to claim 16, wherein the indication of the expected syntax includes a plurality of parameter names and, for each parameter name in the plurality of parameter names, a plurality of candidate parameter values, wherein the output string includes at least one parameter name in the plurality of parameter names and, for each parameter name in the at least one parameter name in the plurality of parameter names, at least one candidate parameter value in the plurality of candidate parameter values.

18. The one or more non-transitory computer-readable storage media of claim 17, wherein the plurality of parameter names correspond to elements that can be created in the special effect creation tool.

19. The one or more non-transitory computer-readable storage media of claim 15, wherein providing the one or more commands to the special effects creation tool comprises: An application programming interface API function provided by the special effect creation tool is called to create the element.

20. The one or more non-transitory computer-readable storage media of claim 15, wherein the special effects creation tool creates the element as a selectable element for application to stored or real-time video content.

Images