Method and apparatus for automatically generating content through visual coding

Abstract

The present specification relates to a method for generating content through visual coding by a recommendation device, and the method may comprise the steps of: receiving an input of a sentence for generating the content from a user; on the basis of the sentence, generating prompt data for configuring an operation in an asset by using a large-scale language model; generating operation configuration data by inputting the prompt data to the large-scale language model; and configuring the operation in the asset on the basis of the operation configuration data.

Description

Method and device for automatically generating content through visual coding

[0001] This specification relates to a method and device for automatically generating content without requiring specialized knowledge of visual coding.

[0002] With traditional development methods, users had to invest significant time and effort in acquiring development knowledge and constantly study new technologies as they emerged. Block coding, a method developed to address these challenges, can help make development more accessible to the general public. Block coding, in particular, has made significant contributions to the education industry and is being utilized in a variety of fields.

[0003] However, block coding also presents challenges for users. For example, users must first understand complex processes, which can require significant learning time. While visual coding can help users acquire coding knowledge and quickly achieve desired results, it still requires users to invest significant time in learning how to use it.

[0004] In particular, users who create content need support that minimizes the time spent learning visual coding and allows them to focus more time on actual content creation. To achieve this, providing a user-friendly environment and intuitive usage methods is crucial. Furthermore, strengthening learning resources and community support is essential to facilitate user acquisition and utilization.

[0005] The purpose of this specification is to provide a method and device for automatically generating content without requiring specialized knowledge of visual coding.

[0006] The technical problems to be solved by this specification are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which this specification belongs from the detailed description of the specification below.

[0007] One aspect of the present specification may include a method for a recommendation device to generate content through visual coding, the method comprising: receiving a sentence for generating the content from a user; generating prompt data for setting an action to an asset using a large-scale language model based on the sentence; generating action setting data by inputting the prompt data into the large-scale language model; and setting an action to the asset based on the action setting data.

[0008] Additionally, the prompt data may include a command corresponding to the sentence.

[0009] Additionally, the above-described operation setting data may include a response word for generating the content based on the above-described command.

[0010] Additionally, the above action may include an animation effect of the above asset.

[0011] In addition, the method may include a step of expanding training data through the large-scale language model; and a step of performing fine-tuning of the large-scale language model using the expanded training data.

[0012] Additionally, the training data may include a set of commands and response words for completing the commands.

[0013] Additionally, the step of expanding the training data may generate similar commands related to commands related to the sentence and generate response words corresponding to the similar commands.

[0014] Another aspect of the present specification is a recommendation device for generating content through visual coding, comprising: a communication unit; a memory including a language model; and a processor for functionally controlling the communication unit and the memory; wherein the processor receives a sentence for generating the content from a user, generates prompt data for setting an action to an asset using the language model based on the sentence, inputs the prompt data to the language model to generate action setting data, and sets an action to the asset based on the action setting data.

[0015] According to embodiments of the present specification, content can be automatically generated without requiring specialized knowledge of visual coding.

[0016] The effects that can be obtained from this specification are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the technical field to which this specification belongs from the description below.

[0017] Figure 1 is a block diagram illustrating an electronic device related to the present specification.

[0018] Figure 2 is an example of an embodiment to which the present specification can be applied.

[0019] Figure 3 is an example of page creation to which this specification can be applied.

[0020] Figure 4 is an example of a controller (400) applicable to this specification.

[0021] Figure 5 is an example of a list of pages to which this specification can be applied.

[0022] Figure 6 is an example of elements to which this specification can be applied.

[0023] Figure 7 is an example of event detection of a fusion manufacturing device to which the present specification can be applied.

[0024] Figure 8 is an example of a result execution method to which this specification can be applied.

[0025] Figure 9 is an example of element management to which this specification can be applied.

[0026] Figures 10 and 11 are examples of element uploads to which the present specification can be applied.

[0027] FIG. 12 is a block diagram of an AI device according to one embodiment of the present specification.

[0028] Figure 13 is an example of a recommended method pipeline to which the present specification can be applied.

[0029] Figure 14 is an example of visual coding to which the present specification can be applied.

[0030] Figure 15 is an example of a recommended device to which the present specification can be applied.

[0031] Figure 16 is an example of a sentence input to which this specification can be applied.

[0032] Figure 17 is another embodiment of a recommended device to which the present specification can be applied.

[0033] Figure 18 is an example of a scene creation method to which the present specification can be applied.

[0034] Figure 19 is an example of prediction of attribute values ​​of assets to which the present specification can be applied.

[0035] Figure 20 illustrates fine-tuning to which the present specification can be applied.

[0036] Figure 21 illustrates a method for setting up an action on an asset to which this specification can be applied.

[0037] Figure 22 illustrates an example of a sentence input to which the present specification can be applied.

[0038] The accompanying drawings, which are incorporated in and constitute a part of the detailed description to aid in understanding the present specification, provide embodiments of the present specification and, together with the detailed description, explain the technical features of the present specification.

[0039] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of this specification.

[0040] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0041] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0042] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0043] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0044] Figure 1 is a block diagram illustrating an electronic device related to the present specification.

[0045] The above electronic device (100) may include a wireless communication unit (110), an input unit (120), a sensing unit (140), an output unit (150), an interface unit (160), a memory (170), a control unit (180), and a power supply unit (190). The components illustrated in FIG. 1 are not essential for implementing the electronic device, and thus the electronic device described in this specification may have more or fewer components than the components listed above.

[0046] More specifically, among the above components, the wireless communication unit (110) may include one or more modules that enable wireless communication between the electronic device (100) and a wireless communication system, between the electronic device (100) and another electronic device (100), or between the electronic device (100) and an external server. In addition, the wireless communication unit (110) may include one or more modules that connect the electronic device (100) to one or more networks.

[0047] This wireless communication unit (110) may include at least one of a broadcast reception module (111), a mobile communication module (112), a wireless Internet module (113), a short-range communication module (114), and a location information module (115).

[0048] The input unit (120) may include a camera (121) or a video input unit for inputting a video signal, a microphone (122) or an audio input unit for inputting an audio signal, and a user input unit (123, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected by the input unit (120) may be analyzed and processed into a user's control command.

[0049] The sensing unit (140) may include one or more sensors for sensing at least one of information within the electronic device, information about the surrounding environment surrounding the electronic device, and user information. For example, the sensing unit (140) may include at least one of a proximity sensor (141), an illumination sensor (142), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor (e.g., a camera (see 121)), a microphone (see 122), a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). Meanwhile, the electronic device disclosed in this specification can utilize information sensed by at least two of these sensors in combination.

[0050] The output unit (150) is for generating output related to visual, auditory, or tactile sensations, and may include at least one of a display unit (151), an audio output unit (152), a haptic module (153), and an optical output unit (154). The display unit (151) may be formed as a layer structure with a touch sensor or formed as an integral part, thereby implementing a touch screen. This touch screen may function as a user input unit (123) that provides an input interface between the electronic device (100) and the user, and at the same time, provide an output interface between the electronic device (100) and the user.

[0051] The interface unit (160) serves as a passageway for various types of external devices connected to the electronic device (100). This interface unit (160) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port. In the electronic device (100), appropriate control related to the connected external device can be performed in response to the external device being connected to the interface unit (160).

[0052] In addition, the memory (170) stores data that supports various functions of the electronic device (100). The memory (170) can store a plurality of application programs (or applications) that run on the electronic device (100), data for the operation of the electronic device (100), and commands. At least some of these application programs can be downloaded from an external server via wireless communication. In addition, at least some of these application programs can exist on the electronic device (100) from the time of shipment for the basic functions of the electronic device (100) (e.g., call receiving and sending functions, message receiving and sending functions). Meanwhile, the application programs can be stored in the memory (170), installed on the electronic device (100), and driven by the control unit (180) to perform the operations (or functions) of the electronic device.

[0053] In addition to the operations related to the above application program, the control unit (180) typically controls the overall operation of the electronic device (100). The control unit (180) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above or by operating an application program stored in the memory (170).

[0054] In addition, the control unit (180) can control at least some of the components discussed with reference to FIG. 1 to drive an application program stored in the memory (170). Furthermore, the control unit (180) can operate at least two or more of the components included in the electronic device (100) in combination to drive the application program.

[0055] The power supply unit (190) receives external power and internal power under the control of the control unit (180) and supplies power to each component included in the electronic device (100). The power supply unit (190) includes a battery, and the battery may be a built-in battery or a replaceable battery.

[0056] At least some of the above components may cooperate with each other to implement the operation, control, or control method of the electronic device according to the various embodiments described below. In addition, the operation, control, or control method of the electronic device may be implemented on the electronic device by driving at least one application program stored in the memory (170).

[0057] In this specification, the electronic device (100) may include a recommendation device, a terminal, a visual coding device, and a fusion manufacturing device.

[0058] Figure 2 is an example of an embodiment to which the present specification can be applied.

[0059] Referring to Figure 2, a user can communicate with a convergence production device via a terminal. For example, the terminal can connect to the convergence production device via the web without a separate application, and the user can simultaneously produce 2D and 3D content via the terminal.

[0060] The recommendation device includes an AI device (20) described below, is connected to the user's terminal and the fusion production device, and uses meta information generated from the terminal, platform, and fusion production device to analyze the user's tendencies, and utilizes the analysis results to recommend content, elements (Assets), and templates suitable for the user.

[0061] The recommendation device can assign weights to content activities used by users through the platform, analyze information on content created through the fusion production device, extract characteristics, and use them as learning data for the recommendation algorithm.

[0062] Additionally, the recommendation device can analyze information on the content the user is currently working on through the fusion production device and extract real-time characteristics.

[0063] Additionally, the recommendation device can generate JSON sentence data based on sentences entered by the user in the input field via the terminal, utilizing a large-scale language model framework. For example, the recommendation device can extract various keywords by processing the sentences entered by the user using a large-scale language model framework.

[0064] Using this, the recommendation device can recommend content, assets, and templates to the integrated production device, and templates and content to the platform. Furthermore, the recommendation device can analyze uploaded files, generate JSON sentence data through a language model framework, and store this data in asset metadata.

[0065] The fusion production device receives content creation commands from a terminal via the web (S2010). For example, the content may include 2D and / or 3D objects.

[0066] The convergence production device creates pages for content creation (S2020). For example, pages can be expressed in 2D or 3D formats, and the convergence production device can configure the screen by adding and arranging predefined elements (Assets) or templates to each page. More specifically, events can be registered for the added elements, allowing for additional interaction with content users. This allows users to create immersive and creative content.

[0067] Figure 3 is an example of page creation to which this specification can be applied.

[0068] Referring to FIG. 3, a user can receive a page display screen (300) from a convergence production device via a terminal. For example, a single piece of content may include one or more pages. Furthermore, the user can change the page format to 2D or 3D via a layout selection window (310) that may appear on the page display screen (300), and can separately add a virtual space with special functions, such as an AR mode, according to the content user's needs. The convergence production device can register a separate controller (400) depending on the page format. Furthermore, the user can change the size and ratio of the page via the layout selection window (310).

[0069] Referring again to FIG. 2, the fusion manufacturing device registers a controller (400) based on the page (S2030). For example, the fusion manufacturing device may register a controller (400) capable of controlling and interacting with elements based on the page's form.

[0070] Figure 4 is an example of a controller (400) applicable to this specification.

[0071] Referring to FIG. 4, when a user selects an element through a terminal, properties corresponding to the element are displayed in a property window (410), and the fusion production device displays a registered controller (400), and the user can easily edit the properties using a mouse or touch using the controller (400). For example, the user can precisely modify the property values ​​of the element by entering an exact numerical value in the property window (410). The user can perform additional connected functions by adding a tab to the property window (410). For example, the additional connected functions may include the source of the element's creator or media playback information.

[0072] Referring again to FIG. 2, the fusion production device places elements on a page (S2040). For example, depending on the page format, the fusion production device may place predefined elements / templates and / or additionally uploaded elements.

[0073] Figure 5 is an example of a list of pages to which this specification can be applied.

[0074] Referring to Figure 5, one content is composed of a bundle of multiple pages (screens), and a 2D or 3D screen can be selected according to the user's needs.

[0075] For example, a page contains (1) page properties, (2) an event list, and (3) a resource list. More specifically, the event list contains information about events assigned to elements, and the resource list contains information about elements added to the page.

[0076] Figure 6 is an example of elements to which this specification can be applied.

[0077] Referring to Figure 6, the fusion production device can provide users with predefined elements / templates based on page format via a terminal. Additionally, users can upload and arrange additional elements.

[0078] Referring again to FIG. 2, the fusion manufacturing device modifies the attribute values ​​of the element (S2050).

[0079] For example, when a user selects an element placed on a page display screen (300), the fusion production device displays a property window (410) corresponding to the element, and the user can modify the property value by using a mouse, touch, numerical input, etc., using a controller (400) registered based on the page.

[0080] The fusion production device registers events corresponding to elements (S2060). For example, an event may include a set of "actions" and "results." More specifically, an action may define the conditions under which the event occurs. For example, an "action" is a condition for a function to be performed and may include various types of events or calls, such as keyboard events, mouse / touch events, gesture events, area events, value events, and call events.

[0081] Additionally, the results may include “features” and “targets”.

[0082] More specifically, a “function” can define property changes and specific actions to be performed on the “target” that is the purpose of the function when an event is activated, and can include basic properties of elements such as position, size, rotation, and transparency, as well as control functions for media elements such as view, hide, play, stop, and pause. It can also include the user terminal’s camera, GPS, and accelerometer to utilize information from the external environment.

[0083] The fusion production device registers actions, functions, and / or targets corresponding to an event (S2070). For example, the fusion production device may register actions, functions, and / or targets based on the attribute values ​​of an element.

[0084] The fusion production device executes a result when an event for an element is detected based on a registered action (S2080).

[0085] Figure 7 is an example of event detection of a fusion manufacturing device to which the present specification can be applied.

[0086] Referring to Figure 7, the fusion fabrication device can detect events by identifying events corresponding to elements, monitoring the events, and executing the corresponding functions. When an event is detected, the fusion fabrication device can identify properties and execute the results based on the functions and properties to execute the corresponding functions.

[0087] Figure 8 is an example of a result execution method to which this specification can be applied.

[0088] Referring to FIG. 8, the fusion production device can execute “results” simultaneously, and lower “results” can be connected to upper “results” and executed continuously. In addition, there is no limit to the connection of “results”, and the execution result of the upper “result” can be used to perform a continuous function in the next step. Unlike an execution method that is simply driven by a single timeline, this result execution method can provide the user with an environment identical to an actual programming technique in relation to the operation of elements, and can help in naturally learning the programming environment.

[0089] Figure 9 is an example of element management to which this specification can be applied.

[0090] Referring to Fig. 9, since 2D elements and 3D elements have different properties, it is difficult for the fusion production device to control them in the same way. Therefore, the fusion production device first wraps the 2D elements and 3D elements through an object called a "basic element", and manages the elements so that they can be used by expanding them into "use elements" used in the authoring tool of the fusion production device based on the "basic elements". For example, the function of the "use elements" can be configured to control and use the properties of the "primitive elements". More specifically, in Fig. 9, the "use elements" are exemplified as images, videos, shapes, and 3D models, but any form that is advantageous for controlling and displaying the properties of 2D elements and 3D elements can be displayed as a "use element."

[0091] Figures 10 and 11 are examples of element uploads to which the present specification can be applied.

[0092] Referring to FIGS. 10 and 11, the fusion production device (or platform) loads an uploaded element file through a loader and registers an object of the loaded element file in a "use element" that can be corresponding to the loader. The fusion production device can place the uploaded element using the registered "use element."

[0093] Referring to FIG. 10, if the uploaded element file is an image file, the fusion manufacturing device can load the image file through the image loader and register the object of the loaded image file as an image element.

[0094] Referring to FIG. 11, if the uploaded element file is a 3D model file, the fusion manufacturing device can load the 3D model file through a 3D loader, create an object of the loaded 3D model file, create an animation, add the animation to the created object, and register the object as a 3D element.

[0095] For example, if the uploaded element file is a video file, the fusion production device can create an HTMLVideoElement, add it to the screen, and then register the object in the "use element" to control it.

[0096] A single piece of content can contain multiple pages, and each page can contain a "resourceManager" to manage elements and an "eventManager" to manage action events.

[0097] Elements placed on a page have separate depths, so the fusion production device can adjust the depth according to the user's needs and adjust the order in which elements are displayed on the screen.

[0098] For example, when an element is placed in a fusion production device, it registers the element in the Resource Manager of the corresponding page, and can manage it through changes in its properties, status, etc., or through registration and deletion. The Resource Manager not only manages the placed elements, but also generates events for changes, and can call connected functions according to resource changes based on the resource list.

[0099] When an element is added, the fusion fabrication device adds it to the management list via the controller registered on the page, allowing the controller to recognize and manage it as a controllable element. Elements registered to the controller can have their properties modified or controlled using the user's mouse, gestures, touch, or an external controller.

[0100] Additionally, Redo / Undo functionality may be required during the user's control of an element. To achieve this, the fusion authoring device can be configured to record changes to the state of the corresponding element by saving changes made by the controller in the page's History Manager. Upon request, the History Manager can retrieve the element's properties from a list of saved changes and update the current element properties to restore or update the element's state. Since the History Manager saves changes for each element, memory issues can arise. Therefore, to prevent this, the fusion authoring device can limit the number of saved change lists depending on the situation.

[0101] Additionally, the converged production device can designate users to share content during production, allowing them to work simultaneously. For example, when content is initially created, the converged production device can create a unique channel corresponding to the content. The converged production device can designate shared users who can use the unique channel, and when those users access the content, they can be added to the same channel.

[0102] Users on the same channel can exchange changes in real time and synchronize authoring data in real time. To achieve this, communication can be done in real time via websocket or webrtc.

[0103] However, when a new user accesses unsaved content, the content will be synchronized to the previous version, potentially resulting in version discrepancies. To prevent this, when a new user is added to a shared channel and accesses the content for the first time, an initial synchronization can be performed by a specific user among the existing users, who will perform a comprehensive update of any changes made. After this initial synchronization, the converged production device can resolve synchronization issues with unsaved content by sharing data on changes in real time.

[0104] FIG. 12 is a block diagram of an AI device according to one embodiment of the present specification.

[0105] The AI ​​device (20) may include an electronic device including an AI module capable of performing AI processing, a server including the AI ​​module, or the like. In addition, the AI ​​device (20) may be included as a component of at least a portion of the electronic device (100) illustrated in FIG. 1 and may be configured to perform at least a portion of the AI ​​processing.

[0106] The above AI device (20) may include an AI processor (21), a memory (25) and / or a communication unit (27).

[0107] The above AI device (20) is a computing device capable of learning a neural network, and can be implemented as various electronic devices such as a server, desktop PC, laptop PC, tablet PC, etc.

[0108] The AI ​​processor (21) can learn a neural network using a program stored in the memory (25). For example, the AI ​​processor (21) can create a Recurrent Neural Networks model utilizing Tensorflow on the memory (25), and can train this artificial intelligence model using data that can be collected from a user terminal and a fusion manufacturing device.

[0109] For example, a trained AI model may include a language model and generate sub-models with tasks such as:

[0110] 1. Content / Template Recommendation Model: Recommends / generates customized content and templates to users by utilizing data analyzed from content usage and content creation information and a user characteristic model.

[0111] 2. Element Recommendation Model: Customized asset recommendations to content creators by utilizing user characteristic models and asset metadata.

[0112] 3. User Characteristics Analysis Model: Analyze user characteristics through analysis of user learning interest, participation, immersion, user usage, and production data.

[0113] Figure 13 is an example of a recommended method pipeline to which the present specification can be applied.

[0114] Referring to FIG. 13, the artificial intelligence model (1300) can be trained using data collected from the platform and the fusion manufacturing device.

[0115] For example, the platform can be provided to terminals via the web and connected to a convergence production device, providing users with a content creation environment. Furthermore, it can display content created by other users and provide a search function for this content. Furthermore, the platform and convergence production device can provide users with templates that can be used in the content creation environment and provide a search function for this.

[0116] The trained artificial intelligence model can generate 1) a user characteristic analysis model, 2) a content / template recommendation model, and 3) an element recommendation model. The user characteristics output by the user characteristic analysis model can be used for training the artificial intelligence model (1300).

[0117] Additionally, the content / template recommendation model can receive sentences from users, convert them into JSON-format sentence data, and provide automatically generated content and templates to users through the platform and integrated production device.

[0118] Additionally, the element recommendation model can recommend assets to users through a fusion production device using sentence data in JSON format.

[0119] Figure 14 is an example of visual coding to which the present specification can be applied.

[0120] Referring to FIG. 14, a user can communicate with a visual coding device via a terminal. For example, a fusion manufacturing device may include a visual coding device.

[0121] Additionally, the terminal can be connected to a visual coding device via the WEB without a separate application, and users can create content through visual coding via the terminal.

[0122] The terminal creates a page for visual coding and places elements (asset) on the page (S1410). For example, the terminal may display a list of elements and / or a template containing the elements to the user. The user can select elements to be placed on the page from the template.

[0123] The terminal sets the target element for visual coding based on the placed elements (S1420). For example, a user can select a target element by clicking on it among the elements placed on the page.

[0124] The terminal sets user actions related to user interaction (S1430). For example, a user action may be a condition for an event to occur in a target element.

[0125] The terminal sets a result related to the target element based on the user action (S1440). For example, the result may refer to an action performed in relation to the target element when the terminal receives input from the user corresponding to the user action. The result may include a "function" that controls the size, position, and status of the target element, an "operation" that operates variables related to the target element, and a "function page" that indicates page movement.

[0126] The terminal displays the result of the target element based on the user action input (S1450). For example, if the user inputs a single click on the target element, the terminal may display a screen that moves the target element for one second or moves to a different page as a result.

[0127] Figure 15 is an example of a recommended device to which the present specification can be applied.

[0128] Referring to Figure 15, the recommendation device is connected to a user terminal and a convergence production device, learns from collected data, and provides predicted output values ​​from the learned AI model to the platform and convergence production device. For example, the convergence production device may include an authoring tool for content creation.

[0129] The recommendation device receives sentences from the user via the convergence production device and / or platform (S1510). For example, the user may input sentences about a specific topic or story for content creation into the convergence production device. More specifically, the user may input the sentence "Blue sky and grass with a smiling sun rising" into the convergence production device. In this case, the input language may primarily be Korean, and the recommendation device may store the input sentences for subsequent processing.

[0130] Figure 16 is an example of a sentence input to which this specification can be applied.

[0131] Referring to Figure 16, a user can select a chat icon (1610) for entering sentences on the content creation screen. Upon receiving the chat icon (1610), the integrated production device can activate the chat function. Through the terminal, the user can enter sentences for content creation through the displayed chat window (1620).

[0132] Referring again to Figure 15, the recommendation device converts the input sentence into English (S1520). If the input sentence is in a language other than English, the input sentence may be converted into English to ensure the accuracy of the language model included in the recommendation device.

[0133] For example, the sentence "Blue sky and grass with a smiling sun" can be converted to the English sentence "Blue sky and grass with a smile."

[0134] The recommendation device converts sentences into story-like text through a language model (S1530). More specifically, the language model's template can be configured to issue the command "Write a synopsis about the "blue sky and grass with a smile" for the sentence "Blue sky and grass with a smile."

[0135] For example, "Blue sky and grass with a smile"

[0136] In the enchanting world of "Blue Sky and Grass with a Smile," we are transported to a serene and picturesque landscape where the beauty of the blue sky and lush green grass becomes a catalyst for transformation. The story revolves around Emily, a young girl grappling with the hardships of life and burdened by sorrow. However, one fateful day, she stumbles upon a hidden meadow, bathed in the golden sunlight, where the sky is a vivid shade of blue and the grass dances with an infectious joy.

[0137] As Emily immerses herself in this magical haven, she starts to notice how the vibrant hues of the sky and grass affect her mood. The brilliant blue sky acts as a gentle reminder to look beyond her troubles and find hope, while the cheerful green grass inspires her to embrace the present moment and seek happiness. Slowly but surely, a smile starts to grace Emily's face as she discovers the power of perspective and the healing balm of nature.

[0138] With each visit to the meadow, Emily's connection to the blue sky and grass deepens. She begins to cultivate a profound sense of gratitude for the beauty surrounding her, finding solace and strength in the simplicity of life's wonders. The radiant smile that forms on her lips becomes a beacon of optimism, influencing those around her and spreading contagious joy.

[0139] Through this poignant journey, “Blue Sky and Grass with a Smile” highlights the transformative power of nature and the profound impact it can have on our emotional well-being. It reminds us to find solace in the smallest of moments, to appreciate the beauty that surrounds us, and to nurture a smile that can illuminate even the darkest of days.

[0140] This can be converted into a story-like text. This is to compensate for the fact that the sentences entered by the initial user lack information for content generation, templates, and / or asset recommendations.

[0141] The recommender uses a language model to transform a story-like text into a plot, including the setting, main character, and key elements (S1540). The recommender can then analyze the story-like text and summarize it into a single scene, including the setting, main character, and key elements, to create a plot. This is intended to extract core elements of the content. The recommender can extract the core elements—the setting, main character, and key elements—from the vast amount of data contained in the story-like text and generate a plot.

[0142] For this purpose, the template of the language model can be set to Summarize the plot in three lines, including the background, main characters, and main elements.

[0143] For example, the story-type text exemplified in S1530 mentioned above

[0144] In "Blue Sky and Grass with a Smile," Emily, a young girl burdened by sorrow, discovers a hidden meadow where the vibrant blue sky and dancing green grass become catalysts for transformation. Immersed in this magical haven, she learns to find hope, embrace the present, and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to appreciate life's simple wonders.

[0145] It can be converted into a plot called .

[0146] The recommendation device converts the plot into sentence data for content, template, and / or asset recommendations (S1550). If necessary, the recommendation device can translate the plot back into Korean. For example, the plot illustrated in S1540 above is

[0147] In "With the Smile of the Sky and Grass," Emily, a girl burdened by grief, discovers a hidden meadow where vibrant blue skies and dancing green grass become the starting point of change. Immersed in this magical place, she finds hope, embraces the present, and learns to cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauty in life.

[0148] Additionally, the recommendation device can use a language model to convert the plot into JSON format sentence data based on the original words and English words, classification and zero classification, importance, and part of speech.

[0149] Table 1 below provides examples of sentence data.

[0150] {"original":"Blue Sky and Grass with a Smiling Sun""sentence": "In Sky and Grass's Smile, Emily, a girl burdened by grief, discovers a hidden meadow where vibrant blue skies and dancing green grass become the starting point of change. Immersed in this magical place, she learns to find hope, embrace the present, and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauties of life.","words": [{"text": "sky and","english": "sky and","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "grass with","english": "grass with","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "smile and","english": "smile and","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "with a smile","english": "with a smile","classification": "adverb","english_classification": "adverb","importance": 2,"part_of_speech": "MAG"},{"text": "by sorrow","english": "by sorrow","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "heavy","english": "burdened","classification": "adjective","english_classification": "adjective","importance": 3,"part_of_speech": "VA"},{"text": "girl","english": "girl","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "Emily","english": "Emily,","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNP"},{"text": "vibrant","english": "vibrant","classification": "adjective","english_classification": "adjective","importance": 3,"part_of_speech": "VA"},{"text": "Blue","english": "blue","classification": "Adjective","english_classification": "adjective","importance": 3,"part_of_speech": "VA"},{"text": "Sky and","english": "sky","classification": "Noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "Dancing","english": "dancing","classification": "Verb","english_classification": "verb","importance": 3,"part_of_speech": "VV+ETM"},{"text": "Blue","english": "green","classification": "Adjective","english_classification": "adjective","importance": 3,"part_of_speech": "VA"},{"text": "Solution","english": "grass","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "of change","english": "transformation","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "starting point","english": "catalysts for","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "being","english": "become","classification": "Verb","english_classification": "verb","importance": 3,"part_of_speech": "VV+ETM"},{"text": "Hidden","english": "hidden","classification": "Adjective","english_classification": "adjective","importance": 3,"part_of_speech": "VA"},{"text": "Meadow","english": "meadow","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "Discovers","english": "discovers","classification": "Verb","english_classification": "verb","importance": 3,"part_of_speech": "VV+EC"},{"text": "this","english": "this","classification": "pronoun","english_classification": "pronoun","importance": 1,"part_of_speech": "MM"},{"text": "magic","english": "magical","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "like","english": "like","classification": "adjective","english_classification": "adjective","importance": 3,"part_of_speech": "VA"},{"text": "Where","english": "place","classification": "noun","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "immersed,","english": "immersing","classification": "verb","english_classification": "verb","importance": 2,"part_of_speech": "VV+EC"},{"text": "she","english": "she","classification": "pronoun","english_classification": "pronoun","importance": 2,"part_of_speech": "NP"},{"text": "hope","english": "hope","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "looking for","english": "finds","classification": "verb","english_classification": "verb","importance": 2,"part_of_speech": "VV+EC"},{"text": "the present","english": "the present","classification": "noun","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "accepting","english": "accepting","classification": "verb","english_classification": "verb","importance": 2,"part_of_speech": "VV+EC"},{"text": "thank you","english": "gratitude","classification": "noun","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "Cultivating","english": "cultivating","classification": "Verb","english_classification": "verb","importance": 2,"part_of_speech": "VV+ETM"},{"text": "Law","english": "the way","classification": "Noun","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "Learn","english": "learn","classification": "Verb","english_classification": "verb","importance": 2,"part_of_speech": "VV+EC"},{"text": "Become","english": "become","classification": "Verb","english_classification": "verb","importance": 2,"part_of_speech": "VV+EC"},{"text": "nature's","english": "nature's","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "healing power","english": "healing power","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "through","english": "through","classification": "investigation","english_classification": "particle","importance": 1,"part_of_speech": "JKM"},{"text": "Emily's","english": "Emily's","classification": "noun","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "shining","english": "shining","classification": "adjective","english_classification": "adjective","importance": 3,"part_of_speech": "VA"},{"text": "smile","english": "smile","classification": "noun","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "optimism","english": "optimism's","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "상징이","english": "symbol","classification": "명사","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "되어","english": "becomes","classification": "동사","english_classification": "verb","importance": 2,"part_of_speech": "VV+EC"},{"text": "그","english": "that","classification": "대명사","english_classification": "pronoun","importance": 1,"part_of_speech": "NP"},{"text": "주변","english": "surrounding","classification": "명사","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "사람들에게","english": "to people","classification": "명사","english_classification": "noun","importance": 2,"part_of_speech": "NNG+JKO"},{"text": "영감을","english": "inspiration","classification": "명사","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "주며,","english": "and","classification": "접속사","english_classification": "conjunction","importance": 1,"part_of_speech": "JC"},{"text": "To us","english": "to us","classification": "Pronoun","english_classification": "pronoun","importance": 2,"part_of_speech": "NP+JKO"},{"text": "Life's","english": "life's","classification": "Noun","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "Simple","english": "simple","classification": "Adjective","english_classification": "adjective","importance": 3,"part_of_speech": "VA"},{"text": "Beauty","english": "beauty","classification": "Noun","english_classification": "noun","importance": 2,"part_of_speech": "NNG"},{"text": "Realize","english": "realize","classification": "Verb","english_classification": "verb","importance": 2,"part_of_speech": "VV+EC"},{"text": "Remind","english": "remind","classification": "Verb","english_classification": "verb","importance": 2,"part_of_speech": "VV+EC"},{"text": "Gives","english": "us","classification": "Pronoun","english_classification": "pronoun","importance": 1,"part_of_speech": "NP"}]},

[0151] Referring to Table 1, the recommendation device can convert the synopsis of the initial input sentence into JSON data. The JSON data can be used to recommend content, templates, and / or assets. This JSON data can include sentence data information such as background, main character, and key elements. Based on the sentence data, the recommendation device recommends content, templates, and / or assets (S1560).

[0152] Figure 17 is another embodiment of a recommended device to which the present specification can be applied.

[0153] Referring to FIG. 17, the recommendation device can generate a list of assets, contents, or templates by inputting keywords (e.g., text fields) included in the sentence data in FIG. 15 described above into the keyword matching system.

[0154] The recommendation device generates keywords for creating a list of recommended assets, content, or templates based on a sentence input by the user (S1710). For example, the recommendation device may receive a sentence for recommending assets, content, or templates from the user via a terminal, and generate keywords for recommending the assets, content, or templates based on the operations of FIG. 15.

[0155] Based on keywords, a list of assets, content and / or templates that can be recommended to users is generated (S1720).

[0156] Table 2 below provides an example list of assets, content, and / or templates that may be recommended to users.

[0157] {"original": "Blue Sky and Grass with a Smiling Sun","sentence": "In With the Smile of the Sky and Grass, Emily, a girl burdened by grief, discovers a hidden meadow where vibrant blue skies and dancing green grass become the starting point of change. Immersed in this magical place, she learns to find hope, embrace the present, and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauty in life","contentList": ["a7e884bb-3d4c-46e2-9cf9-640460eb44d0","a7e884bb-3d4c-46e2-9cf9-640460eb44d0","a7e884bb -3d4c-46e2-9cf9-640460eb44d0","a7e884bb-3d4c-46e2-9cf9-640460eb44d0",...],"TemplateList": ["a7e884bb-3d4c-46e2-9cf9-640460eb44d0","a7e884bb-3d4c-46e2-9cf9-640460eb44d0"," a7e884bb-3d4c-46e2-9cf9-640460eb44d0","a7e884bb-3d4c-46e2-9cf9-640460eb44d0",...],"assetList": [{"assetIdx": "0b70763b-4e22-43bb-a163-68c80f9b9e37","metaData": {"words": [{"text": "Sky and","english": "sky and","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "grass","english": "grass with","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"},{"text": "smile","english": "smile and","classification": "noun","english_classification": "noun","importance": 3,"part_of_speech": "NNG"}, … ]}},{"assetIdx": "0b70763b-4e22-43bb-a163-68c80f9b9e37",.

[0158] Referring to Table 2, the list that can be recommended to the user may include a sentence entered by the user, a story based on the sentence, a content list, a template list, and / or an asset list, and the asset list may include metadata of the asset.

[0159] Figure 18 is an example of a scene creation method to which the present specification can be applied.

[0160] Referring to FIG. 18, the recommendation device can automatically generate a scene included in the content by using the list of assets, content, and / or templates that can be recommended to the user in Table 2 described above.

[0161] The recommendation device generates a list of keywords based on the plot (S1810). For example, the keywords may include the keywords in Table 1 described above (e.g., "text"). These keywords may be generated based on "original" or "sentence." If the keywords are generated based on "sentence," the recommendation device can analyze the "sentence" field using an artificial intelligence model to identify the intent and components of the sentence.

[0162] Table 3 below provides an example list of keywords.

[0163] {"original": "Blue Sky and Grass with a Smiling Sun","sentence": "In With the Smile of the Sky and Grass, Emily, a girl burdened by grief, discovers a hidden meadow where vibrant blue skies and dancing green grass become the starting point of change. Immersed in this magical place, she learns to find hope, embrace the present, and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauty in life.","sentenceKeywordList": [{"keyword": "sky","position": "top","size": "large","layer": "back"},{"keyword": "grass","position": "bottom","size": "middle","layer": "front"},...]}

[0164] Referring to Table 3, the recommender can use a large-scale language model to extract keywords from a "sentence" and predict attribute values ​​(e.g., position, size, layer) for each keyword to generate a list of keywords. In more detail, the recommender can extract keywords from a "sentence" and predict attribute values ​​for each keyword to generate a "sentenceKeywordList."

[0165] In addition, the recommender can extract keywords for "original" in the same way and predict attribute values ​​for each keyword to create a "sentenceKeywordList." More specifically, the recommender can linguistically analyze the "sentence", which is a modified version of the input user sentence "blue sky and grass with a smiling sun rising," and extract the keywords "sky" and "grass." The recommender can analyze the context of the "sentence" to predict attribute values, such as "position," "size," and "layer," for each keyword, and record them in the "sentenceKeywordList."

[0166] For example, the position can be distinguished as "upper" and "lower", the size can be distinguished as "large", "medium" and "small", and the order can be distinguished as "front", "back" and "middle".

[0167] A list of these keywords can be combined with a list of recommended assets to produce data in JSON format.

[0168] Table 4 below illustrates the format of data combining a list of keywords and a list of recommended assets.

[0169] {"original": "...","sentence": "...","sentenceKeywordList": [{"keyword": "...","position": "...","size": "...","layer": "..."},...],"recommendAssetList: [{"assetIdx": "...","metaData": {"words": [{"text": "...","english": "...","classification": "...","english_classification": "...","importance": "...","part_of_speech": "..."},...],}},...]}

[0170] Referring to Table 4, the "assetList" item in Table 2 can be managed as included in the recommendAssetList item. The recommendation device matches keywords with assets based on the list of keywords and the list of recommended assets (S1820). For example, the recommendation device can use a language model to compare "keyword" with the "text" in the "metadata" item of "recommendAssetList," and match keywords with assets based on a linguistic similarity level greater than a certain value.

[0171] Table 5 below illustrates the matched results.

[0172] {"original": "...","sentence": "...","processedData": [{"matchingKeywordData": {"keyword": "...","position": "...","size": "...","layer": "..."},"matchingAssetList": [{"assetIdx": "..."...},...]},...]}

[0173] Referring to Table 5, the recommendation device can manage keywords and assets matched by pairs in the "processedData" array. Based on the attribute values ​​of the matched keywords, the recommendation device predicts the attribute values ​​of the corresponding assets (S1830). For example, the recommendation device can predict the attribute values ​​of assets matched with the keywords using a numerical prediction model based on the "position," "size," and "layer" attribute values ​​of the included keywords included in the "matchingKeywordData" item.

[0174] Figure 19 is an example of prediction of attribute values ​​of assets to which the present specification can be applied.

[0175] Referring to FIG. 19, the recommendation device can predict the attribute values ​​of the corresponding asset based on the attribute values ​​of the matched keyword.

[0176] Additionally, the recommendation device can access information about content using the corresponding asset through the database and extract the corresponding asset's attribute values ​​from the content. In this case, the recommendation device can filter the asset attribute values ​​in the content using the corresponding asset based on the attribute values ​​of the matching keyword, and utilize the filtered valid data to predict the asset's attribute values.

[0177] Table 6 below shows examples of predicted asset attribute values.

[0178] {"data": [{"sentenceKeyword": "...","assetList": [{"assetIdx": "...","predictedProperty": {"x": "...","y": "...","z": "...","depth": "...","width": "...","height": "...",...}},...]},...]}

[0179] Referring to Table 6, the recommender can use a numerical prediction model to predict the coordinates, depth, width, and height within the page for the "predictedProperty" property corresponding to the asset. The recommender can utilize these properties to visually represent or arrange the asset. For example, it can place the asset in a specific location on the page or use height and depth to place keywords in 3D space.

[0180] Again, referring to FIG. 18, the recommendation device adds recommended assets to the page based on the recommended asset list (S1840). For example, the recommendation device may add assets with a high priority from the recommended asset list to the page.

[0181] The recommendation device sets predicted attribute values ​​for the added assets (S1850). This allows the recommendation device to place assets in appropriate locations on the page and create scenes for content generation.

[0182] Figure 20 illustrates fine-tuning to which the present specification can be applied.

[0183] The recommendation device can set an animation effect for the recommended assets during the process of recommending assets based on sentences entered by the user.

[0184] Referring to Figure 20, a large-scale language model can generate a scene by setting actions on assets based on a user-entered sentence. For this purpose, the large-scale language model can be fine-tuned.

[0185] The recommendation device expands the training data through a large-scale language model (S2010).

[0186] For example, training data may include commands related to sentences entered by the user and responses used to complete those commands. Each training data set may consist of instructions for a specific command and a corresponding response.

[0187] More specifically, if the command is "Rotate 90 degrees", the response might be composed as {'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} .

[0188] Also, if the command is "Go to page 3", the response could be composed as {'action':{'type':'actionActionsClick'},'function':{'type':'page-move','value':['3']}}.

[0189] Through this, large-scale language models can be trained to output responses to set actions on assets in response to user-entered sentences.

[0190] Table 7 below shows an example of predicted training data.

[0191] [{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 90도 회전해줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 3페이지로 이동해줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'page-move','value':['3']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 확대해줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionScale','value':['*2']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request.### Instruction: 왼쪽으로 이동 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionMove','value':['left']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 오른쪽으로 이동해줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionMove','value':['right']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 작게 줄여줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionScale','value':[' / 2']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 재생시켜줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionPlay'}} / ""},{"text": "Below is an instruction that describes a apoc studio task.Write a response that appropriately completes the request. ### Instruction: 일시정지시켜줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionStop'}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 숨겨줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionHide'}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 보여줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionShow'}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 180도 회전시켜줘 ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['180']}} / ""}].

[0192] However, since this training data does not cover the vast amount of similar sentences that users can input, the commands need to be expanded. To achieve this, the recommender can expand the training data using a large-scale language model. For example, the recommender can generate similar commands related to commands through a large-scale language model and expand the training data with pairs of responses to these commands.

[0193] Table 8 below illustrates the extended training data.

[0194] [{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Rotate 90 degrees ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a Write a response that appropriately completes the request. ### Instruction: Rotate the photo 90 degrees clockwise. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate':['90']}} / ""},{"text": "Below is an instruction that describes a response that appropriately completes. the request. ### Instruction: Please rotate the image counterclockwise. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Please rotate the image clockwise.### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes an apoc studio task. Write a response that appropriately completes the request. ### Instruction: Please display the image rotated 90 degrees. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Please rotate the image so that it is horizontal for better viewing. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Rotate the photo 90 degrees to the right.### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Could you rotate the image 90 degrees counterclockwise? ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Please rotate the image clockwise and then display it. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Display the image in a new perspective, rotated 90 degrees.### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Could you rotate the image to the right? ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Rotate the image counterclockwise and show the result. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Rotate the photo 90 degrees clockwise and then show it.### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Could you please rotate the image counterclockwise? ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Please display the image in its new form, rotated 90 degrees. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Rotate the photo 90 degrees to the right.### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 이미지를 시계 방향으로 돌려 주실 수 있을까요? ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: 그림을 90도 돌린 후에 결과를 확인해 주세요. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request.### Instruction: Could you please rotate the picture counterclockwise? ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Please display the image rotated 90 degrees to the right. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},{"text": "Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Please display the picture from a new perspective, rotated 90 degrees. ### Response: / "{'action':{'type':'actionActionsClick'},'function':{'type':'functionRotate','value':['90']}} / ""},.

[0195] The recommendation device fine-tunes a large-scale language model using the expanded training data (S2020). For example, the recommendation device can fine-tune a large-scale language model using the expanded training data. This allows the recommendation device to create a large-scale language model to set actions for recommended assets based on sentences entered by the user.

[0196] Figure 21 illustrates a method for setting up an action on an asset to which this specification can be applied.

[0197] Referring to FIG. 21, the large-scale language model may include a model fine-tuned by the aforementioned FIG. 20.

[0198] The recommendation device receives a sentence from the user (S2110). For example, the recommendation device may receive a sentence from the user through the aforementioned S1510 operation. More specifically, the recommendation device may receive the sentence "Blue sky and grass with a smiling sun rising" from the user.

[0199] Figure 22 illustrates an example of a sentence input to which the present specification can be applied.

[0200] Referring to FIG. 22, a user may select an asset (2200) and input a sentence to set an action for the asset through a sentence input window (2210).

[0201] The recommendation device generates prompt data for setting actions using a large-scale language model based on the input sentence (S2120).

[0202] For example, prompt data may be generated based on an "original" (the original sentence entered by the user) or may be generated based on a "sentence" (the story) described above.

[0203] Table 9 below illustrates prompt data.

[0204] Original Rising Sun Prompt Data Below is an instruction that describes a apoc studio task. Write a response that appropriately completes the request. ### Instruction: Rising Sun ### Response:

[0205] Referring to Table 9, if the user inputs "rising sun," the recommendation device may target the "sun" asset and, using a large-scale language model, generate prompt data to configure an action. For example, the prompt data may include an instruction corresponding to the sentence entered by the user. The recommendation device inputs the prompt data into the large-scale language model to generate action configuration data (S2130). For example, the action configuration data may include a response corresponding to the instruction.

[0206] Table 10 below illustrates the motion configuration data.

[0207] Sleep response"{ / 'action / ':{ / 'type / ': / 'actionActionsClick / '}, / 'function / ':{ / 'type / ': / 'functionMove / ', / 'value / ':[ / 'up / ']}}"

[0208] Referring to Table 10, the generated motion setting data is exemplified based on the prompt data exemplified in Table 9. The response included in this motion setting data can be in a form that can be directly applied to the authoring tool. The recommendation device sets an action for an asset based on the motion setting data (S2140). For example, the recommendation device can analyze a sentence entered by a user and apply the response to the acquired target asset (e.g., the sun) to set an action. Alternatively, if the user selects an asset and inputs a sentence, the recommendation device can set an action for the asset.

[0209] The recommendation device can automatically generate a scene containing assets with set actions through the aforementioned actions.

[0210] This specification allows users to easily create content that includes assets with applied motions, such as animation effects, simply by entering sentences.

[0211] The above-described specification can be implemented as computer-readable code on a program-recorded medium. A computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also includes media implemented in the form of carrier waves (e.g., transmission via the Internet). Therefore, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of this specification should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalency range of this specification are intended to be included within the scope of this specification.

[0212] In addition, while the above description focuses on services and embodiments, these are merely examples and do not limit the present specification. Those skilled in the art to which this specification pertains will appreciate that various modifications and applications not exemplified above are possible without departing from the essential characteristics of the present service and embodiments. For example, each component specifically shown in the embodiments can be modified and implemented. In addition, differences related to such modifications and applications should be construed as being included within the scope of this specification as defined in the appended claims.

Claims

1. In the method of generating content through visual coding by the recommendation device, A step of receiving a sentence for generating the above content from a user; A step of generating prompt data for setting an action on an asset using a large-scale language model based on the above sentence; A step of inputting the above prompt data into the above large-scale language model to generate operation setting data; and A step of setting an action to the asset based on the above action setting data; A method of producing a product, comprising:

2. In paragraph 1, The above prompt data is A generation method comprising a command corresponding to the above sentence.

3. In paragraph 2, The above operation setting data is A method of generating content, comprising a response word for generating the content based on the above command.

4. In paragraph 3, The above action A method of creating an asset, comprising an animation effect of the above asset.

5. In paragraph 1, Through the above large-scale language model, a step of expanding the training data; and A step of performing fine-tuning of the large-scale language model using the extended training data; A method of producing a product, comprising:

6. In paragraph 5, The above training data is A method of generating a command, comprising: a set of commands; and a set of response words for completing the commands.

7. In paragraph 6, The step of expanding the above training data is A method of generating similar commands related to a command related to the above sentence, and generating response words corresponding to the similar commands.

8. In a recommendation device that generates content through visual coding, Department of Communications; A memory containing a language model; and A processor for functionally controlling the above communication unit and the above memory; The above processor A recommendation device that receives a sentence for generating the content from a user, generates prompt data for setting an action for an asset using the language model based on the sentence, inputs the prompt data into the language model to generate action setting data, and sets an action for the asset based on the action setting data.

Images