Information processing system, information processing method, and program

Abstract

This invention provides an information processing system, method, and program for generating multimodal content. [Solution] The information processing system 90 includes a policy generation function 930 that generates a generation policy 200 including constraints 210 and performance policies 220 for generating multimodal content based on input information relating to the subject of the multimodal content 800 to be generated; an element generation function 940 that generates at least two pieces of information from visual information 300, audio information 400 and text information 500 of the multimodal content based on the generation policy; an editing function 950 that generates multimodal content by editing at least two pieces of information to conform to the constraints based on the generation policy; and a command function 960 that manages the execution status of generating the generation policy, generating at least two pieces of information from visual information, audio information and text information, and generating the multimodal content.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Patent Document 1 describes "a system including means for uploading a video file, means for inputting text data specifying a specific scene, means for analyzing the video file based on the input text data using a generation AI model to detect a specific scene, means for extracting the detected scene to generate a highlight video, and means for providing the generated highlight video to a user". [Prior Art Document] [Patent Document] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2026-035284

Summary of the Invention

Means for Solving the Problems

[0003] According to one embodiment of the present invention, an information processing system for generating multimodal content is provided. The information processing system may include a policy generation function that generates a generation policy, which includes constraints for generating the multimodal content and a performance policy, which includes performance guidelines for generating the multimodal content, based on input information relating to the subject of the multimodal content to be generated. The information processing system may include an element generation function that generates at least two pieces of information from among visual information relating to the visual elements of the multimodal content, audio information relating to the audio elements, and text information relating to the text elements, based on the generation policy. The information processing system may include an editing function that generates the multimodal content by editing the at least two pieces of information according to the constraints, based on the generation policy. The information processing system may include a command function that manages the execution status of the generation of the generation policy by the policy generation function, the generation of the at least two pieces of information from among the visual information, audio information, and text information by the element generation function, and the generation of the multimodal content by the editing function. The information processing system may include a plurality of functions, including these.

[0004] In the information processing system, the policy generation function may generate the generated policy by adjusting the theatrical guidelines to the extent that the constraints are met.

[0005] In any of the aforementioned information processing systems, the constraints may include a format policy that defines constraints on the format of the multimodal content, and a content policy that defines constraints on the content that the multimodal content should include. In any of the aforementioned information processing systems, the policy generation function may generate the generation policy by adjusting the content policy to the extent that the format policy is satisfied, and by adjusting the presentation policy to the extent that both the format policy and the content policy are satisfied.

[0006] In any of the aforementioned information processing systems, the constraints may include a formal policy that defines constraints on the formal aspects of the multimodal content and a content policy that defines constraints on the content that the multimodal content should include. In any of the aforementioned information processing systems, the policy generation function may generate the generation policy by managing it to prohibit the generation of the content policy and the production policy until the generation of the formal policy is complete, and to prohibit the generation of the production policy until the generation of the content policy is complete.

[0007] In any of the aforementioned information processing systems, the element generation function may include a format policy among the generation policy that defines the constraints on the format of the multimodal content in the generated product obtained by generating at least two of the visual information, audio information, and text information.

[0008] In any of the aforementioned information processing systems, the input information may include intent information representing the intention to produce the multimodal content and material information relating to the materials of the multimodal content. In any of the aforementioned information processing systems, the policy generation function may generate the generation policy based on the intent information and the material information.

[0009] In any of the above information processing systems, the input information may include intent information representing the intention to produce the multimodal content and images relating to the materials of the multimodal content. Any of the above information processing systems may further have an analysis function that analyzes the images and obtains structured text data of the images. In any of the above information processing systems, the policy generation function may generate the generation policy based on the intent information and the structured text data of the images.

[0010] In any of the aforementioned information processing systems, the policy generation function may generate the generation policy while managing it to prohibit the generation of the generation policy until the analysis function has finished acquiring the structured text data of the image.

[0011] In any of the aforementioned information processing systems, the input information may include images relating to the materials of the multimodal content. In any of the aforementioned information processing systems, the policy generation function may generate the generation policy which includes a policy prohibiting at least one of the subject attributes included in the image, such as shape, color, components, logo, and price, from changing in a manner contrary to natural laws between the start and end points of the multimodal content.

[0012] In any of the above information processing systems, the input information may further include content that serves as a reference for generating the multimodal content. In any of the above information processing systems, the policy generation function may generate the generation policy based on the reference content. In any of the above information processing systems, the element generation function may generate at least two of the visual information, audio information, and text information based on the reference content.

[0013] Any of the aforementioned information processing systems may include an information receiving function for receiving the input information from a user. Any of the aforementioned information processing systems may include a change management function for managing whether the input information received by the information receiving function, the generated policy generated by the policy generation function, the at least two pieces of information generated by the element generation function, and the multimodal content generated by the editing function can be modified.

[0014] In any of the aforementioned information processing systems, the command function may manage the start and completion of the execution of the plurality of functions.

[0015] In any of the aforementioned information processing systems, the command function may initiate the execution of the policy generation function, determine the execution order of the plurality of functions based on the generated policy generated by the policy generation function upon completion of the execution of the policy generation function, initiate the execution of one of the plurality of functions according to the determined execution order, and initiate the execution of the next function in the determined execution order upon completion of the execution of that one function.

[0016] Any of the above information processing systems may further include a quality check function that outputs a content evaluation result by performing at least one of the following evaluations: evaluating the product generated by the policy generation function based on the input information; evaluating the product generated by the element generation function based on at least one of the input information and the generation policy; and evaluating the product generated by the editing function based on at least one of the input information and the generation policy. In any of the above information processing systems, the command function may adjust the execution status based on the content evaluation result.

[0017] In any of the above information processing systems, the multimodal content may be video content including the visual information, the audio information, and the text information. In any of the above information processing systems, the element generation function may have a storyboard generation function that generates storyboard information from the visual information. In any of the above information processing systems, the element generation function may have a video generation function that generates video information from the visual information. In any of the above information processing systems, the element generation function may have a background music (BGM) generation function that generates background music from the audio information. In any of the above information processing systems, the element generation function may have a narration generation function that generates narration from the audio information. In any of the above information processing systems, the input information may include intent information that represents the intention to produce the video content and material information relating to the materials of the video content. In any of the above information processing systems, the command function may manage the start and completion of execution of multiple functions provided by the information processing system. In any of the above information processing systems, the policy generation function may generate the generation policy based on the intent information and the material information in response to the execution start process by the command function. In any of the above information processing systems, the command function may, upon completion of the execution of the policy generation function, determine the execution order of a plurality of functions, including the storyboard generation function, the video generation function, the BGM generation function, and the narration generation function, based on the generation policy generated by the policy generation function, and manage to start the execution of one of the plurality of functions according to the determined execution order, and to sequentially execute the process of starting the execution of the next function in the determined execution order upon completion of the execution of that one function.

[0018] In any of the above information processing systems, the input information may include intent information that represents the intention to produce the multimodal content. In any of the above information processing systems, the policy generation function may convert the intent information expressed from the perspective of the producer of the multimodal content into information expressed from the perspective of the viewer of the multimodal content, and generate the generation policy based on the information expressed from the viewer's perspective.

[0019] In any of the aforementioned information processing systems, the element generation function may, based on the generation policy, classify the emotional elements to be included in the multimodal content into multiple energy levels, and based on the energy level, control at least one of the volume, tempo, and beat of the audio information, as well as the presence or absence of a human voice.

[0020] In any of the aforementioned information processing systems, the element generation function may generate background music (BGM) to be included in the audio information, based on the generation policy, under the constraint that it constitutes a single, consistent BGM from the start to the end of the multimodal content.

[0021] Any of the aforementioned information processing systems may include a plurality of AI (Artificial Intelligence) agents. In any of the aforementioned information processing systems, the plurality of AI agents may realize the plurality of functions. In any of the aforementioned information processing systems, the AI ​​agent that realizes the command function and the AI ​​agent that realizes the element generation function may be different from the plurality of AI agents.

[0022] In any of the aforementioned information processing systems, the policy generation function, the element generation function, the editing function, and the command function may each be implemented by an AI agent. In any of the aforementioned information processing systems, the AI ​​agents that implement the policy generation function, the element generation function, the editing function, and the command function may be different from each other.

[0023] According to an embodiment of the present invention, there is provided an information processing method for generating multimodal content, which is executed by a computer. The information processing method includes generating a generation policy including a production policy regarding constraints in the generation of the multimodal content and a production policy regarding dramatic guidelines in the generation of the multimodal content based on input information regarding the theme of the multimodal content to be generated by a policy generation function; generating at least two pieces of information out of visual information regarding visual elements of the multimodal content, audio information regarding audio elements, and text information regarding text elements based on the generation policy by an element generation function; and generating the multimodal content by an editing function that edits the at least two pieces of information so as to comply with the constraints based on the generation policy, and may include a command stage for managing the execution status.

[0024] According to an embodiment of the present invention, there is provided a program for causing a computer to execute any of the above information processing methods.

[0025] Note that the above summary of the invention does not list all of the necessary features of the present invention. Also, sub-combinations of these feature groups can also be inventions.

Brief Description of Drawings

[0026] [Figure 1] Schematically shows an example of the prior art. [Figure 2] Schematically shows an example of an information processing system 90. [Figure 3] Schematically shows an example of an information processing system 90. [Figure 4] Schematically shows an example of an information processing system 90. [Figure 5] Schematically shows an example of an information processing system 90. [Figure 6] An example of processing performed by the information processing system 90 is shown in outline. [Figure 7] An example of the information processing system 90 is shown in outline. [Figure 8] An example of the processing flow by the information processing system 90 is shown in outline. [Figure 9] A schematic example of the hardware configuration of a computer 1200, which functions as a server on which an AI agent is implemented to realize various functions such as an information processing device 900, a storage device 909, and a command function 960, is shown. [Modes for carrying out the invention]

[0027] The present invention will be described below through embodiments of the invention, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0028] Traditionally, the production of multimodal content such as short videos was done manually. Each stage—planning, filming, BGM production, and editing—was handled by specialists in each field. However, manual production is time-consuming and expensive. For example, sharing intentions and transferring materials between specialists in each stage requires significant communication costs and time. Furthermore, revisions necessitate going back to earlier stages, leading to prolonged production periods. Therefore, producing multimodal content quickly and at low cost has been difficult.

[0029] In recent years, attempts have been made to generate multimodal content 800 using AI (Artificial Intelligence). For example, a user can input instructions into a single generating AI to generate multimodal content 800. Conventional AIs may ignore pre-imposed rules or produce hallucination, which fabricates details of information. In some cases, multiple AIs are combined to generate multimodal content 800. In this case, the user must direct specialized AIs, such as the AI ​​responsible for video production and the AI ​​responsible for audio production, adjusting the consistency between the generated videos and audio to improve the overall quality of the multimodal content 800. Such direction and adjustment require specialized knowledge and effort. Therefore, for example, when generating short videos to advertise a specific product, it has been difficult to consistently generate high-quality content that does not damage the brand image of that product.

[0030] The information processing system 90 according to this embodiment has a configuration that contributes to solving such problems. For example, the information processing system 90 has a function that mimics and executes the production flow of each specialist for multiple processes of generating multimodal content 800, and an orchestration function that manages the execution status of those functions. Each function may be implemented by a specialized AI agent. For example, the information processing system 90 receives the materials for the multimodal content 800 and user requests as input information 100, and causes an AI agent acting as a scriptwriter to generate a generation policy 200. A specialized AI agent that produces video and audio executes its respective assigned process in strict compliance with the generation policy 200. The orchestration function of the information processing system 90 dynamically manages the execution status of each AI agent based on the generation policy 200. In this way, multimodal content 800 is generated while ensuring the consistency of information between each element by generating and editing components such as video and background music. This allows us to provide users with multimodal content of the same quality as that produced by experts, without relying on individual user operations on each AI.

[0031] Figure 1 schematically shows an example of conventional technology. In the example shown in Figure 1, the conventional process for producing multimodal content 800 is explained. In this example, a human video production team performs the production of multimodal content 800 in multiple stages. For example, the video production team sequentially performs the following stages from the first to the fifth: planning and composition, storyboard creation, video shooting, BGM creation, and editing.

[0032] In the example shown in Figure 1, each stage—planning, filming, background music production, and editing—is handled by specialists in their respective fields. However, manual production requires the time and expense of each individual responsible. Furthermore, sharing production intentions and exchanging materials among the different teams incurs communication costs and time.

[0033] In the example shown in Figure 1, manual production requires going back to previous stages and redoing the work if a correction is needed at any stage, which leads to a prolonged production period. Therefore, for example, it was difficult to produce multimodal content 800 at low cost and in a short period of time.

[0034] In recent years, AI has been used to generate multimodal content. For example, a user might input abstract instructions to a single generating AI to generate content in bulk. However, conventional AIs sometimes ignored pre-defined rules or generated hallucinations that fabricated details of information.

[0035] In some cases, multiple AIs are combined to generate multimodal content. For example, based on the user's expertise in operating each AI, the user gives instructions to each AI to generate visual information such as storyboards and videos, audio information such as background music and narration, and text information such as captions. In such cases, the user themselves needs to adjust the consistency between these elements. Therefore, for example, when generating short videos to advertise a specific product, it has been difficult to consistently generate high-quality content that does not damage the brand image of that product.

[0036] Figure 2 schematically shows an example of an information processing system 90. In the example shown in Figure 2, the input information 100 that is input to the information processing system 90 and the multimodal content 800 that is generated by the information processing system 90 based on the input information 100 will be explained.

[0037] In the example shown in Figure 2, the input information 100 is information about the subject of the multimodal content 800 to be generated. The subject may be the core elements of the production, such as the intention behind the creation of the multimodal content 800, such as advertising, or information about the specific product, service, or store that it targets, such as "what to express, with what intention, and using what materials." The input information 100 may include, for example, intention information 110 that represents the intention behind the content's creation, and material information 120 that constitutes the content's components. In this example, we will explain using the case of generating an advertising video for the ramen shop "Jikasei Men Shiro to" and its product "Tanrei Shio Soba Platinum" as the multimodal content 800.

[0038] The subject of Multimodal Content 800 may include goods, services, and stores. For example, the subject may be ramen. The subject may be the provision of food and beverages. The subject may be a ramen shop. The subject may be the purpose of Multimodal Content 800. For example, the subject may be advertising.

[0039] In the example shown in Figure 2, intent information 110 is the highest priority constraint that binds the entire process of creating the multimodal content 800, and may include supreme provisions, like a constitution, that define the requirements that the multimodal content 800 must satisfy, the elements that should be included in its content, or matters that should be avoided. Details of the constraints will be described later. Intent information 110 may include, for example, mandatory requirements for content creation. Mandatory requirements may be matters that users want to be reflected in the multimodal content 800. In this example, the mandatory requirements are to express "a live taste experience: the change in taste due to 'yuzu espuma (foam)' and the aroma of truffle oil" in the multimodal content 800, and to convey "consideration for the body: being completely free of chemical additives and using whole wheat flour in the homemade noodles (guilt-free)." Furthermore, the required specifications may include details regarding the format of the multimodal content 800, such as "a video in mp4 format with a resolution of 1080px x 1920px and a frame rate of 30fps."

[0040] Intent information 110 may include, for example, prohibited items in content creation. Prohibited items may be things that users should definitely avoid in multimodal content 800. In this example, the prohibited items are to avoid "volume appeals: expressions that appeal to fullness such as 'free extra serving' and 'hearty portion'" in multimodal content 800, and to not include "richness: words that evoke heaviness such as 'rich' and 'extra fat' (because the product is marketed as 'light' and 'flavorful')." Thus, in this example, intent information 110 specifies as a must-achieve item the emphasis on the change in flavor due to the yuzu espuma (foam) that is a characteristic of "Tanrei Shio Soba Platinum," and specifies as prohibited items expressions that appeal to fullness such as "free extra serving" and expressions that evoke heaviness such as "rich," which contradict the branding of "Jikasei Men Shiro to."

[0041] In the example shown in Figure 2, the material information 120 includes specific information about the product being advertised. In this example, the material information 120 is information about "Tanrei Shio Soba Platinum," the signature dish of the ramen restaurant "Jikasei Men Shirato." The material information 120 includes multiple images, such as "Product Image 1," which shows the appearance of the product; "Product Image 2," which conveys the texture of the soup and espuma (foam); "Image of the interior of the store"; and "Image of the exterior of the store." The material information 120 may also include the store name, store characteristics, product name, product characteristics, service name, service characteristics, product price, and service price.

[0042] In the example shown in Figure 2, the store's characteristics are described as follows: "We aim to 'redefine ramen as a modern meal,' and we adhere to a completely additive-free approach, relying solely on the power of the ingredients to create delicious flavor without using chemical seasonings. We combine the skills of our craftsmen with meticulous calculations, such as making our own noodles from domestic wheat in a glass-walled noodle-making room and using delicate cooking techniques from French cuisine. We offer a 'clean taste' that you can eat every day without feeling guilty, and that your body will actually appreciate, all through uncompromising handiwork." In this example, the product description is: "A luxurious bowl of ramen with a golden broth made from the umami of whole Ooyama chicken, clams, and Rausu kelp, infused with the aroma of white truffle oil. Our homemade whole wheat noodles, made with domestic wheat 'Haru yo Koi,' are characterized by their fragrant aroma and smooth texture. By dissolving the 'yuzu espuma (foam)' floating in the center as you eat, you can enjoy a dramatic change in taste, from rich umami to refreshing citrus flavor."

[0043] In the example shown in Figure 2, the information processing system 90 comprehensively processes the various input information formats 100 described above and outputs multimodal content 800. In this example, the output multimodal content 800 is, for example, a vertical short video of about 15 seconds, suitable for advertising media such as social media.

[0044] In the example shown in Figure 2, the multimodal content 800 is configured so that multiple elements generated based on the input information 100 are synchronized and coordinated. For example, at the beginning of the video, an appealing image of "Tanrei Shio Soba Platinum" is displayed, the product's features are read aloud by a narrator, the product name and price extracted from the input information 100 are superimposed in appropriate positions within the video, and background music appropriate to the overall atmosphere is superimposed.

[0045] In the example shown in Figure 2, the generated multimodal content 800 strictly adheres to the constraints described in the intent information 110. In this example, the essential element of "flavor change" is visually and audibly emphasized in the video, while prohibited keywords such as "extra large portion" do not appear at all. This allows, for example, users to quickly obtain advertising videos based on sophisticated brand strategies simply by providing input information 100, without having to edit individual materials.

[0046] In the example shown in Figure 2, the specific elements included in the multimodal content 800 will be explained. The multimodal content 800 may include visual information 300, audio information 400, and text information 500. In this example, the visual information 300 constituting the multimodal content 800 includes a video of ramen and a video of the restaurant interior.

[0047] In the example shown in Figure 2, the video 320 of the ramen, which serves as visual information 300, is a visual representation of the product's features with added staging. In this example, video 320 may include close-up footage of the yuzu espuma (foam) dissolving into the soup. Video 320 may also include slow motion, for example. This allows viewers to intuitively understand the product's delicate texture and live feel, stimulating their appetite and interest.

[0048] In the example shown in Figure 2, the in-store video 320, as visual information 300, is a video intended to convey the atmosphere of the store. In this example, video 320 may include cooking scenes and customer seating scenes. For example, video 320 may include scenes of a cook carefully draining noodles and customers enjoying their meal in the calm atmosphere of the store. In other words, video 320 may dynamically visualize not only the physical attributes of the subject but also contextual information such as the serving process and surrounding environment, and may encompass video elements that appeal to the multifaceted value and reliability of the subject through the viewer's simulated experience. This allows for, for example, a multifaceted appeal to the store's commitment and sense of security, rather than simply introducing food.

[0049] In the example shown in Figure 2, background music (BGM) 410 may be included as audio information 400 constituting the multimodal content 800. In this example, the BGM 410 is selected to be a calm and relaxing piece of music that matches the store's concept of "redefining modern dining." For example, sophisticated jazz or ambient music centered on acoustic instruments might be selected. In other words, the audio information 400 may dynamically auditorily represent not only the subject's emotional attributes but also the subject's worldview and the passage of time, and may include audio elements that complement the subject's brand value. This can, for example, create a sense of luxury and cleanliness, thereby improving the brand image.

[0050] In the example shown in Figure 2, the text information 500 that constitutes the multimodal content 800 may include multiple captions. In this example, the text information 500 is displayed sequentially according to the viewer's psychological changes and the characteristics of the product. In the example shown in Figure 2, at the beginning of the video, the caption "Is this ramen...?" is displayed as a hook to attract the viewer's interest. Next, the caption "First, eat it as is, then dissolve the foam" is displayed to guide how to eat the product. Furthermore, the caption "A bowl of ramen that your body will love, completely free of artificial additives" is displayed to explain the added value of the product. Finally, the caption "Homemade Noodles Shiroto," which is the name of the store, is displayed.

[0051] In the example shown in Figure 2, information regarding the distinctiveness of the subject may be displayed. For example, a design that enhances the distinctiveness of the brand may be displayed. For example, a brand logo may be displayed. A mark, trademark, etc. corresponding to the subject may be displayed.

[0052] In the example shown in Figure 2, the video, background music 410, and on-screen text (text information 500) are arranged in a highly synchronized manner to maintain information consistency. In this example, for example, when the on-screen text "Dissolve the bubbles" is displayed, a video of bubbles actually dissolving may be played, and sound effects that emphasize this change may be superimposed. This allows the appeal of the product and the brand message to be deeply imprinted in the viewer's memory, even with short viewing times.

[0053] Figure 3 schematically shows an example of an information processing system 90. In the example shown in Figure 3, the logical relationships of the multiple functions of the information processing system 90 are schematically shown. In this example, the information processing system 90 includes a policy generation function 930, an element generation function 940, an editing function 950, and a command function 960. The multiple functions of the information processing system 90 may be able to share information with each other via a communication channel 97.

[0054] The communication channel 97 may include at least one of a network connecting multiple devices and an internal communication channel such as a software bus within the same device. The network may include a mobile communication network. The network may conform to 5G (5th Generation). The network may conform to LTE (Long Term Evolution). The network may conform to 6G (6th Generation) or later communication systems. The network may include the internet. The network may include the cloud. The network may include a LAN (Local Area Network).

[0055] In the example shown in Figure 3, each function of the information processing system 90 may be implemented by AI. For example, each function may be implemented by a generative AI. Each function may be implemented by multiple generative AIs. Each function may be implemented by an AI agent. Each function may be implemented by multiple AI agents. Some or all of the functions of each may be implemented by different separate AI agents. Each function may be implemented by a single AI agent. Some of the functions of each may be implemented as an AI agent, and the remaining functions may be implemented by a program that is not an AI. The program may be implemented by an API (Application Programming Interface) specialized for a particular process. Each AI agent may use an API appropriate to the purpose of the function.

[0056] The generative AI may be an AI model capable of generating elements included in multimodal content 800 such as text, images, videos, and audio. The generative AI may be, for example, a language model. For example, the generative AI may be a large language model (LLM). The generative AI may be, for example, a small language model (SLM). The generative AI may be a language model of an intermediate size between an SLM and an LLM. The generative AI may be, for example, a diffusion model capable of generating images and videos. For example, the generative AI may be a model that targets content generation in general, such as a speech synthesis model that generates audio. The generative AI may be a Transformer model. Note that the generative AI may be composed of a combination of multiple types of models.

[0057] In the example shown in Figure 3, the relationship between each function and the information that each function inputs and outputs will be explained. The policy generation function 930 generates a generation policy 200 based on the input information 100 input to the information processing system 90. The generation policy 200 includes constraints 210 for the generation of multimodal content 800 and a performance policy 220 concerning performance guidelines.

[0058] In the example shown in Figure 3, the generation policy 200 may be a structure of instructions in a format that can be interpreted by subsequent element generation functions 940 and editing functions 950, which incorporate the user's subjective intentions and material characteristics contained in the input information 100. In this example, the constraints 210 may be standards that must be adhered to in the multimodal content 800. For example, the constraints 210 may include the physical specifications of the multimodal content 800. For example, the physical specifications may include the length of the multimodal content 800, the aspect ratio of the video, the frame rate, the resolution, the number of gradations, etc. The constraints 210 may include the content requirements of the multimodal content 800. The constraints 210 may include the content prohibitions of the multimodal content 800. In addition, the generation policy 200 may include other policies such as guidelines for ensuring consistency between elements. For example, the generation policy 200 may include a policy that functions as a blueprint for the temporal overlap of visual information 300, audio information 400, and text information 500.

[0059] In the example shown in Figure 3, the production policy 220 may be a guideline for production aimed at enhancing the quality and atmosphere of the multimodal content 800. Production refers to expressive techniques for bringing out the appeal of the material, and may involve emphasizing and visually realizing emotional values ​​such as the texture and live feel of the subject through control of the camera angle, adjustment of playback speed, etc. For example, the production policy 220 may specify directions for the production of the multimodal content 800, such as "maintain a sophisticated, clean, and modern tone overall," "use calm jazz-style background music," and "place text overlays neatly, making good use of white space."

[0060] If the policy generation function 930 is an AI agent, the AI ​​agent that implements the policy generation function 930 may include prompts that define the design logic for formulating a generation policy 200, which is a blueprint for multimodal content 800, from the input information 100. These prompts may include the following logical components:

[0061] The policy generation function 930 may include a logical component relating to the absolute hierarchical structure of rules. For example, such a logical component may include provisions for autonomously adjusting multiple instructions or constraints according to a predefined priority order when they conflict with each other. Specifically, for example, such a logical component may include definitions of hierarchies, in order from top to bottom, such as physical and formal constraints, content requirements, interpretive and creative guidelines. For example, if a specific phrase specified in the content requirements exceeds the character limit of the physical and formal constraints, the policy generation function 930 will perform adjustments such as splitting and line breaks according to the "form" of the higher-level rule while maintaining the "content" of the lower-level rule, and will prohibit alteration of the information through summarization or paraphrasing.

[0062] The policy generation function 930 may include a logical component relating to a stepwise execution process. This logical component may separate content design into a material analysis phase and a policy formulation phase, and may include a provision prohibiting the transition to the policy formulation phase until the execution of the tools in the preceding analysis phase and the understanding of the material are completely finished. This structurally suppresses the occurrence of hallucinations based on misunderstandings or unverified information about the material.

[0063] The policy generation function 930 may include a logical component relating to the definition of direction based on creator attributes. This logical component may include a provision for selecting the optimal one from several predefined creator attributes based on the brand characteristics and target audience included in the input information 100. Creator attributes include, for example, trendsetter, expert, and storyteller. Based on the selected attribute, the policy generation function 930 may define the narration style, BGM tempo, and video cut rhythm in a consistent tone and output them as a direction policy 220.

[0064] The policy generation function 930 may include a logical component for calculating multimodal synchronization parameters. Multimodal synchronization parameters may be time-series control indicators for precisely matching video, audio, and text elements along a common time axis. For example, multimodal synchronization parameters may include a timestamp specifying a point in time, an offset specifying a deviation from a reference point, a duration specifying a period, and cue points specifying triggers. The logical component may include information for defining the time allocation to each scene of the multimodal content 800 and the timing of events occurring between media. For example, the logical component may include provisions for allocating the total playback time (e.g., 15 seconds) to each scene and defining the video direction, text overlay content, and sound effects along the time axis in each time slot. For audio elements, multiple energy levels representing emotional fluctuations may be defined, and control indicators for synchronizing visual changes in video with auditory effects may be generated.

[0065] Thus, the policy generation function 930, by incorporating a hierarchical rule compliance logic and a step-by-step design process, can formulate a generation policy 200 that balances strict constraints to ensure brand credibility with creative presentations that appeal to the audience's sensibilities.

[0066] The element generation function 940 generates information for constructing multimodal content 800 based on the generation policy 200. The element generation function 940 may generate information for constructing multimodal content 800 in accordance with the performance policy 220 while adhering to the constraints 210. The element generation function 940 generates at least two of the following information: visual information 300 concerning the visual elements of the multimodal content 800, audio information 400 concerning the audio elements, and text information 500 concerning the text elements. In this example, the element generation function 940 generates visual information 300, audio information 400, and text information 500.

[0067] The editing function 950 generates multimodal content 800 by editing at least two of the visual information 300, audio information 400, and text information 500 output by the element generation function 940, based on the generation policy 200. The editing function 950 may edit the at least two pieces of information in accordance with the performance policy 220 while adhering to the constraints 210. In this example, the editing function 950 generates multimodal content 800 by editing the visual information 300, audio information 400, and text information 500.

[0068] If the editing function 950 is implemented by an AI agent, the AI ​​agent may include prompts that define video integration logic for integrating materials such as visual information 300, audio information 400, and text information 500 output from each function to generate the final multimodal content 800. These prompts may include the following logic components:

[0069] The editing function 950 may include a logical component for strict sequential control of joining and compositing. This logical component may include a provision for sequentially processing according to a predefined stepwise pipeline, rather than compositing multiple materials at once. This provision applies a stepwise compositing logic, for example, in which multiple video clips are first joined in the correct order, then narration is superimposed, and finally background music is mixed. This allows for verification of the integrity of intermediate products and the presence or absence of incomplete materials, including anomalies or omissions, at the end of each step. In other words, it is possible to immediately detect if an error occurs in one step and structurally prevent proceeding to the next step with an incomplete product.

[0070] The editing function 950 may include a logical component for verifying the availability of materials necessary for editing and managing the dependencies between each material. This logical component may include a provision to check the status of the workflow managed by the command function 960 before starting processing, and to confirm that all sub-processes for generating necessary elements such as video 320, BGM 410, and narration are complete. If even one required file is missing, the editing function 950 may include a provision to suspend processing and report the type and reason for the missing material to the command function 960.

[0071] Editing function 950 may include a logical component for managing physical assets using absolute paths. This logical component may include a provision that strictly prohibits the use of relative paths when referencing material information and saving intermediate products, and only uses absolute paths (full paths) on storage. This ensures traceability of content generation by eliminating file reference errors and overwrites, even when multiple AI agents are running in a distributed environment.

[0072] The editing function 950 may include a logical component for generating artifacts in multiple output formats and declaring the completion of the final artifact generation. This logical component may include provisions for automatically generating and outputting both a version with superimposed text and a clean version without text, based on the instructions of the generation policy 200. After the editing function 950 reports the save path of the final artifact, it may include provisions for declaring specific control commands to terminate the entire project process and prompting the release of computing resources.

[0073] Thus, the editing function 950, with its video integration logic including strict control over the compositing order, verification of material availability, and asset management using absolute paths, can suppress the effects of hallucination and generate professional-quality multimodal content 800 that maximizes the contributions of each specialist agent.

[0074] The command function 960 manages the execution status of the generation of the generation policy 200 by the policy generation function 930, the generation of each piece of information by the element generation function 940, and the generation of the multimodal content 800 by the editing function 950. The command function 960 may schedule the generation of the multimodal content 800 based on the generation policy 200. For example, the command function 960 may determine the sub-processes that should be included in the process of generating the multimodal content 800, determine the execution order of the determined sub-processes, and determine the functions responsible for each sub-process. The command function 960 may dynamically schedule based on the generation policy 200. The command function 960 may schedule based on the generation policy 200 and predetermined scheduling criteria. For example, the command function 960 may select one scheduling format from several predetermined scheduling formats based on the performance policy 220 included in the generation policy 200. For example, if the production policy 220 is a policy that generates multimodal content 800 including video, audio, and text, and is a policy that places emphasis on video, the command function 960 selects a scheduling format in which the video is generated first, followed by the generation of audio and text. The command function 960 may also cause the policy generation function 930 to determine the order of the sub-processes after policy generation, and then schedule according to the order determined by the policy generation function 930. The command function 960 may also control the execution of the sub-processes after policy generation in a predetermined order based on the generation policy 200. The command function 960 may also control the execution of the sub-processes after policy generation in a manner that is known as batch processing.

[0075] The command function 960 may manage the execution status of the functions responsible for each sub-process so as to strictly follow the determined order of the sub-processes. The command function 960 may manage the start and completion of execution of multiple functions. For example, the command function 960 may determine the completion of one sub-process and manage to start other subsequent sub-processes. The command function 960 may also manage to prohibit the start of other subsequent sub-processes until one sub-process is completed.

[0076] The command function 960 may start the execution of the policy generation function 930, and upon completion of the execution of the policy generation function 930, determine the execution order of multiple functions based on the generated policy 200 generated by the policy generation function 930, start the execution of one of the multiple functions according to the determined execution order, and upon completion of the execution of that one function, start the execution of the next function in the determined execution order.

[0077] In this embodiment, the execution order may be information representing the dependencies between processes. For example, the execution order may be information indicating which process can be started once which process is completed. The execution order may include a serial order, a parallel branching order, an order in which one of several paths is selected based on the result of the previous process, and an order that does not matter (not in any particular order).

[0078] The command function 960 may determine, for example, that policy generation should be executed before the other two processes of policy generation, image generation, and BGM generation. On the other hand, the command function 960 may determine that, for example, in image generation and BGM generation, although both refer to the generation policy 200, these processes can be executed in parallel if there is no physical dependency such as BGM not being able to be created until the image is complete.

[0079] Command function 960 may determine the execution order such that BGM generation comes first if the generation policy 200 prioritizes BGM. Command function 960 may determine the execution order such that narration generation comes first if the generation policy 200 prioritizes the emotional experience.

[0080] The command function 960 may manage so that the next function to be executed can access the information that is the output of processing performed by a particular function, in response to the acquisition of information indicating that the output of processing performed by that function has been output. For example, the command function 960 may use a tool that centrally manages the execution status of each function. If each function records whether or not it has completed its processing in a process management table where the execution status of each function is centrally managed, the command function 960 may acquire information based on the process management table indicating whether or not the processing of a sub-process by a particular function has been completed, and if the information indicates that the processing of that sub-process has been completed, it may manage so that it grants access rights to the information that is the output of that sub-process to the next function to be executed. The command function 960 may manage so that functions other than the next function to be executed cannot access the information that is the output of that function. The command function 960 may also manage whether or not to access the target product by notifying whether or not to notify the absolute path to the storage area where the target product is stored, rather than managing access rights.

[0081] In addition to the command function 960 managing the execution status of each function, each function itself may acquire information indicating whether the processing of a sub-process by the preceding function has been completed, and if the information indicates that the processing of the sub-process has not been completed, it may prohibit itself from starting its own process. In this case, each function may be able to refer to information indicating the execution order of each sub-process determined by the command function 960. This allows each function to autonomously cooperate to generate multimodal content 800 from input information 100.

[0082] If the command function 960 is implemented by an AI agent, the AI ​​agent may include prompts that define decision-making guidelines for coordinating multiple specialist agents. These prompts may include the following logical components:

[0083] Command function 960 may include a logical component relating to the completion constraints of the thinking process. This logical component may include a provision prohibiting the selection of other AI agents to handle the next step until it is objectively confirmed that one AI agent currently in execution has completed all the thought processes defined for its role. This prevents, for example, incomplete information or undefined intermediate products from being propagated to subsequent steps, thereby ensuring the overall integrity of the content.

[0084] Command function 960 may include a logical component relating to workflow process tracking. This logical component may define the entire content generation process as multiple discrete sub-processes, such as planning and design, visual element generation, audio element generation, and final editing, and may include provisions for centrally monitoring the progress status (not started, in progress, completed, failed) of each sub-process. Command function 960 may prohibit the initiation, skipping, and reordering of dependent subsequent stages unless a specific preceding sub-process is recorded as completed.

[0085] Command function 960 may include a logical component relating to role selection logic. This logical component may include provisions for dynamically identifying the most suitable expert agent to speak or act next, based on conversation history, the status of shared workflows, and specific keywords contained in input information 100. Command function 960 may refer to an agent catalog defining the scope of responsibility for each expert agent and grant execution privileges to the agent that best matches the current phase of the project.

[0086] Command function 960 may include logical components for exception handling and termination determination. These logical components may include provisions for determining whether to interrupt processing or initiate a recovery loop to attempt correction when abnormal situations are detected, such as when a specific tool execution fails repeatedly or when required file paths are missing. These logical components may also include provisions for declaring the project finalized only after confirming that all editing and quality checks have been completed.

[0087] Thus, the command function 960 includes a logical component that acts as an orchestrator, not only controlling the execution of individual agents but also strictly monitoring and synchronizing the completion status of each agent's thought process and the progress of the workflow. This allows for the generation of sophisticated multimodal content 800 while suppressing hallucination.

[0088] In the example shown in Figure 3, each function of the information processing system 90 may be implemented on a cloud server on the network, or on a local terminal on the user's device. For example, some of the functions may be implemented on the cloud server and the remaining functions on the local terminal (hybrid type). All functions may be implemented on the cloud server (fully cloud type). All functions may be implemented on the local terminal (fully local type).

[0089] Figure 4 schematically shows an example of the information processing system 90. The example shown in Figure 4 will primarily explain the differences from the example shown in Figure 3. This example illustrates one example of the aforementioned hybrid implementation configuration.

[0090] In the example shown in Figure 4, the information processing system 90 includes an information processing device 900. In this example, of the multiple functions provided by the information processing system 90, the command function 960 is implemented in the information processing device 900. In this example, each of the other functions other than the command function 960 is implemented on a cloud server on the network 99. The information processing device 900 may be the user's local terminal.

[0091] In the example shown in Figure 4, the information processing device 900 is connected to each cloud server via the network 99 in a communicative manner. In this example, the command function 960 implemented in the information processing device 900 instructs each function on the cloud server to perform processing and manages its execution status. As a result, even if the information processing device 900 does not have high computing power, for example, it is possible to generate high-quality multimodal content 800 by utilizing the resources of the cloud server.

[0092] Figure 5 schematically shows an example of the information processing system 90. The example shown in Figure 5 will primarily explain the differences from the example shown in Figure 4. In this example, as a specific example, the process of generating an advertising video for the aforementioned "Homemade Noodles Shiroto" will be used for explanation.

[0093] In the example shown in Figure 5, the information processing system 90 further includes an information receiving function 910, an analysis function 920, a visual information generation function 942, a storyboard generation function 942a, a video generation function 942b, an audio information generation function 944, a background music generation function 944a, a narration generation function 944b, a text information generation function 946, a change management function 970, a quality check function 980, an output control function 990, and a storage device 909. In this example, the input information 100 includes intent information 110, material information 120, and reference content 130. The input information 100 does not necessarily include reference content 130. In this example, the material information 120 includes an image 122 related to the material. In this example, the constraints 210 include a format policy 212 and a content policy 214.

[0094] In the example shown in Figure 5, the information receiving function 910 may receive various types of information. For example, the information receiving function 910 receives input information 100. The information receiving function 910 may be an interface function for receiving input information 100 transmitted from outside the information processing system 90, and input information 100 stored in an external storage device or the like.

[0095] In the example shown in Figure 5, the information receiving function 910 may acquire various data such as material information 120, images 122, and reference content 130 contained in the input information 100 via the network 99. The various information acquired by the information receiving function 910 may be stored in a format that can be used by various functions such as the policy generation function 930, the element generation function 940, or the command function 960. In this example, the information acquired by the information receiving function 910 is stored in the storage device 909.

[0096] The information receiving function 910 may include a web server function, an API, or a file-watching function that detects the storage of files in a specific folder, etc., for accepting uploads from user terminals. The information receiving function 910 may perform preprocessing on the received input information 100, such as format conversion, noise reduction, or metadata addition. Upon receiving the input information 100 from the information receiving function 910, the information processing system 90 may start a series of multimodal content generation processes 800.

[0097] In the example shown in Figure 5, the information reception function 910 may obtain necessary input information 100 from the user by providing a predetermined input form. The information reception function 910 may also include a function as an interactive agent that extracts the optimal input information 100 through dialogue with the user. The information reception function 910 may perform dynamic question-and-answer sessions with the user. The information reception function 910 may perform question-and-answer sessions to improve the quality of the multimodal content 800. For example, based on the product information entered by the user, the information reception function 910 may ask questions such as, "What is the most unique feature of this product's materials that other stores don't have?" to draw out potential appeals and selling points that the user themselves may not be aware of, thereby supplementing the input information 100.

[0098] In the example shown in Figure 5, the information receiving function 910 may include a moderation function to determine the validity and safety of the input information 100. The information receiving function 910 may determine whether the user's input contains excessive exaggeration, information suspected of being false, defamation of others, or expressions that violate public order and morals. If it determines that inappropriate expressions are included, the information receiving function 910 may temporarily withhold the input and, through dialogue, suggest to the user revisions to more sincere, attractive, and respectful expressions, guiding them towards harmless and appropriate input information 100. This facilitates the generation of socially reliable, high-quality multimodal content 800 in the subsequent generation process.

[0099] If the information reception function 910 is implemented by an AI agent, the AI ​​agent may have an input management logic for receiving and structuring the diverse forms of input information 100 provided by the user in a format that can be processed by each specialized agent in the system. This logic may include the following logical components.

[0100] The information receiving function 910 may include a logical component for collecting multiple types of materials and managing them separately. This logical component may include provisions for receiving product and store information in text format (such as a CSV file) and images as visual information 300 in pairs, and for managing each pair by assigning uniquely identifiable information. This provides a foundation that allows subsequent analysis functions 920, for example, to accurately link and analyze which image corresponds to which product attribute.

[0101] The information receiving function 910 may include a logical component for extracting constraints that must be strictly followed. This logical component may include provisions for identifying mandatory and prohibited items that must be strictly followed in content generation from the input information 100, and flagging them as priority policies that are clearly distinguished from other general reference information. This allows for the extraction of strict constraints that serve as the basis for formulating hierarchical rules, for example.

[0102] The information reception function 910 may include a logical component for identifying the selling points and appeal of the target product that are not explicitly stated in the input information, through interaction with the user. This logical component may include provisions for performing dynamic question-and-answer sessions with the user if it determines that the provided initial information is insufficient. For example, by detecting a discrepancy between visual features obtained from a product image (e.g., the transparency of the soup) and text information, and asking a specific question such as, "Is the greatest feature of this product the method used to make this clear soup?", the system can then add and reinforce the input information 100 with potential appeals that the user may not even be aware of, based on the answers obtained.

[0103] The information receiving function 910 may include a logical component for pre-verifying the integrity and security of the input. This logical component may include provisions for verifying, prior to policy generation, whether the input text content contains excessive exaggeration, suspected falsehood, or expressions that violate public order and morals. If inappropriate input is detected, the information receiving function 910 may guide the user towards harmless and appropriate input information 100 by suggesting revisions to more honest and appealing expressions.

[0104] In this way, the information receiving function 910, by incorporating input management logic including structural acceptance of materials, identification of absolute constraints, active information reinforcement, and input guardrails, can, for example, prevent information inconsistencies and quality degradation in subsequent generation processes, and establish a foundation for generating multimodal content 800 that accurately reflects brand value.

[0105] In the example shown in Figure 5, the analysis function 920 may analyze the input information 100. The analysis function 920 may analyze the image 122 related to the material. The analysis function 920 may analyze the image 122 and obtain structured text data 124 of the image 122. The structured text data 124 may be information that describes the names, attributes, states, and relationships between subjects contained in the image in a format that can be interpreted by the subsequent policy generation function 930. For example, the analysis function 920 obtains the structured text data 124 in JSON format. In this example, the policy generation function 930 may generate a generated policy 200 based on the intent information 110 and the structured text data 124 of the image 122.

[0106] In the example shown in Figure 5, the policy generation function 930 may be managed to prohibit the generation of the generation policy 200 until the analysis function 920 has finished acquiring the structured text data 124 of the image. This management may be performed by the command function 960. After the command function 960 has acquired information indicating the completion of processing by the analysis function 920, it may instruct the policy generation function 930 to start generating the generation policy 200.

[0107] In the example shown in Figure 5, for example, the analysis function 920 analyzes the source image 122 (an image of ramen) and obtains structured text data 124 that reads "Name: Shio Soba, Topping: Yuzu Espuma, Bowl: White Porcelain". The policy generation function 930 combines this structured text data 124 with intent information 110, "I want to create a sense of luxury," to generate a generation policy 200 that includes the content "emphasize the texture of the white porcelain bowl and highlight the whiteness of the yuzu espuma." In this way, by structuring the image 122 before generating the policy, the accuracy of information transmission between AI agents is improved, and multimodal content 800 that better reflects the appeal of the source material can be generated.

[0108] If the analysis function 920 is implemented by an AI agent, the AI ​​agent may include prompts that define material understanding logic for structuring the unstructured material data contained in the input information 100 into a format usable in subsequent design processes. These prompts may include the following logical components:

[0109] The analysis function 920 may include a logical component for multimodal semantic integration. This logical component may include provisions for comparing the content of text-based material information 120 and image 122, verifying their consistency, and integrating them into a single structured data. For example, for a product described as "golden soup" in text, the function may analyze whether the color of the soup in the corresponding image matches, and establish that characteristic as a visual fact.

[0110] The analysis function 920 may include a logical component for structuring (metadata extraction) unstructured images. This logical component may include provisions for converting the subject, background, lighting, composition, and color distribution within the image into detailed text data 124 using image analysis tools, multimodal analysis tools, etc. This allows the abstract atmosphere of an image to be converted into concrete attribute information that subsequent agents can logically handle.

[0111] The analysis function 920 may include a logical component for extracting brand identity. This logical component may include provisions for determining the intrinsic value and unique tone of the brand from all provided materials. This allows for the identification of higher-level attributes that guide presentation policies, such as whether the brand is luxurious, traditional, modern, or innovative, based on the quality of the materials and the style of expression, rather than merely extracting keywords.

[0112] Analysis function 920 may include a logical component for gatekeeping the transition to the design phase. This logical component may include a provision that strictly prohibits the commencement of the next step, policy formulation, until all material analysis tools have been executed and it is determined that a sufficient understanding of the material has been obtained. This allows, for example, the occurrence of hallucination based on an insufficient understanding of the material to be suppressed at its source.

[0113] Thus, the analysis function 920, equipped with material understanding logic including the integration of multimodal information, detailed image structuring, identification of brand attributes, and strict gate control between processes, can lead to the development of accurate and sophisticated generation policies 200, for example, to suppress hallucination and maximize the appeal of the input material.

[0114] In the example shown in Figure 5, the generation policy 200 includes constraints 210 and a presentation policy 220. In this example, constraints 210 includes a format policy 212 and a content policy 214. The format policy 212 may define constraints on the format of the multimodal content 800. The content policy 214 may define constraints on the content of the multimodal content 800.

[0115] In the example shown in Figure 5, the policy generation function 930 may generate a generation policy 200 based on intent information 110 and material information 120. The policy generation function 930 may generate the generation policy 200 by adjusting the performance guidelines to the extent that the constraints 210 are satisfied. In this example, the policy generation function 930 generates a performance policy 220 to the extent that the constraints 210 are satisfied.

[0116] In the example shown in Figure 5, if a conflict occurs between the constraints 210 and the performance policy 220, the policy generation function 930 prioritizes the constraints 210 and adjusts the content of the performance policy 220 to generate the generated policy 200. In this example, a conflict between the constraints 210 and the performance policy 220 may include a state in which, if each piece of information is generated or edited according to the performance policy 220, the compliance requirements regarding the format and content of the multimodal content 800 as defined in the constraints 210 cannot be satisfied. If such a conflict is likely to occur, the policy generation function 930 may perform adjustments to the content of the performance policy 220, such as limiting, modifying, and partially deleting it, to the extent that it conforms to the constraints 210.

[0117] For example, suppose that while constraint condition 210's formal policy 212 strictly stipulates that "the playback time of multimodal content 800 shall be 15 seconds or less," the production policy 220 includes a production guideline stating that "to convey the store's commitment, a 10-second slow-motion effect should be applied to both the scene of draining the noodles and the scene of the yuzu espuma (foam) dissolving." In this case, if production policy 220 is executed as is, the slow-motion portion alone would total 20 seconds, exceeding constraint condition 210's 15-second limit, thus causing a conflict. The policy generation function 930 prioritizes constraint condition 210 and dynamically adjusts the content of production policy 220, for example, by "reducing the time of each slow-motion scene to 5 seconds," or by "making only the more important scene a 5-second slow-motion scene, while the other is at normal speed," and outputs the final generation policy 200. This allows, for example, to fully reflect the creative intent behind the advertisements while complying with physical constraints such as ad space regulations and distribution platform specifications.

[0118] In the example shown in Figure 5, the policy generation function 930 may generate the generation policy 200 by adjusting the content policy 214 to the extent that the format policy 212 is satisfied, and by adjusting the presentation policy 220 to the extent that both the format policy 212 and the content policy 214 are satisfied. The generation policy 200 may have a hierarchical decision-making logic with the constraints of the physical format of the multimodal content 800 as the highest priority, followed by required items and prohibited items, and finally creative presentation.

[0119] In the example shown in Figure 5, the policy generation function 930 first fixes the format policy 212 (e.g., resolution, aspect ratio, total duration), which is the highest priority, and then adjusts the content policy 214 (e.g., required appeal items and prohibited expressions) so that it fits within that framework. Furthermore, assuming that both of these are fully satisfied, it optimizes the production policy 220 (e.g., tone, manner, BGM style, camera work). This prevents lower-level production preferences from compromising higher-level content specifications or important advertising content.

[0120] Let's explain a specific example of how the policy generation function 930 generates a generation policy 200. For example, consider a case where a generation policy 200 is initially generated with the following constraints: "Format policy 212: Playback time within 15 seconds, Content policy 214: A 5-second explanation of the owner's commitment as a requirement, a 5-second scene of the yuzu espuma melting, an 8-second scene of the actual eating, and Direction policy 220: Apply cinematic slow motion to all scenes for twice the playback time." The policy generation function 930 then adjusts this policy. In this case, the policy generation function 930 first detects that the total of 18 seconds specified in the content policy 214 does not fit within the 15 seconds specified in the format policy 212, and adjusts the content policy 214. For example, it shortens the low-priority eating scene to 5 seconds to fit the total content into 15 seconds. Next, considering the use of slow motion under these constraints of the format policy 212 and content policy 214, the playback time becomes 30 seconds, which deviates from the format policy 212, so the direction policy 220 is adjusted. Specifically, the adjustment is made to "slow down only the most important espuma scene, and keep everything else at normal speed," and the final generation policy 200 is output. In this way, by performing step-by-step judgments and adjustments in the order of format, content, and direction, it is possible to generate an effective generation policy without failure, even when multiple complex constraints are intertwined.

[0121] In the example shown in Figure 5, the policy generation function 930 may prohibit the generation of content policy 214 and performance policy 220 until the generation of format policy 212 is complete. The policy generation function 930 may prohibit the generation of performance policy 220 until the generation of content policy 214 is complete. Such management may also be performed by the command function 960.

[0122] In the example shown in Figure 5, the element generation function 940 includes a visual information generation function 942 that generates visual information 300, a sound information generation function 944 that generates sound information 400, and a text information generation function 946 that generates text information 500. In this example, the visual information generation function 942 includes a storyboard generation function 942a and a video generation function 942b. In this example, the sound information generation function 944 includes a background music generation function 944a and a narration generation function 944b. In this example, each of these functions may be implemented as an independent AI agent with a specialized role.

[0123] In the example shown in Figure 5, the hierarchical structure of each function of the element generation function 940 is illustrated. However, this merely illustrates the conceptual inclusion relationships of the elements generated by each function for clarity, and it is not necessarily required that these functions themselves be connected in a hierarchical logical structure. For example, in the example shown in Figure 5, the element generation function 940 has a visual information generation function 942, and the visual information generation function 942 is illustrated as having a hierarchical structure that includes a storyboard generation function 942a and a video generation function 942b. However, the storyboard generation function 942a and the video generation function 942b may each be directly connected to the network 99, receive instructions directly from the command function 960, and transmit the generated products.

[0124] In the example shown in Figure 5, the storyboard generation function 942a generates a storyboard 310 that defines the composition, direction, and sequence of each scene constituting the multimodal content 800, based on the generation policy 200. In this example, the storyboard generation function 942a may include not only the "attractiveness of the product" being advertised, but also the "attractiveness of the space" that gives viewers the experience of virtually visiting the store.

[0125] As a specific example in this case, the storyboard generation function 942a defines the following scene composition as storyboard 310, which constitutes the advertising video for "Homemade Noodles Shiro to". First, it defines a "product appeal scene" that shows the overall view of "Tanrei Shio Soba Platinum" at the beginning, and then a close-up of the "yuzu espuma (foam)" dissolving into the soup. Next, the storyboard generation function 942a defines storyboard 310 for the "in-store space scene" that conveys the store's commitment and comfortable atmosphere. Specifically, it defines a "dynamic cut emphasizing craftsmanship" in which the cook carefully drains the domestic wheat noodles in a clean, open kitchen. Furthermore, it may define a "wide shot (long shot) that conveys a modern store image" in which customers calmly enjoy their meal in a sophisticated store space where the glass-enclosed noodle-making room is visible. Finally, it defines an ending cut that displays the store logo. Thus, the storyboard generation function 942a may output a storyboard 310 that is not merely a list of information, but a visual story that structures the sense of confidence in the quality of the product (cooking scene) and the value of visiting the store (customer seating scene).

[0126] In the example shown in Figure 5, the video generation function 942b generates the video 320 that will actually be played, based on the storyboard 310 and the material information 120. In this example, the video generation function 942b uses a generation AI such as a diffusion model to generate the video 320 as dynamic footage of "bubbles dissolving" as specified in the storyboard 310, while maintaining the texture of the images in the material information 120.

[0127] If the storyboard generation function 942a is implemented by an AI agent, the AI ​​agent may include prompts that define visual design logic for formulating a storyboard 310 that precisely defines the visual composition of each scene based on the generation policy 200. These prompts may include the following logic components:

[0128] The storyboard generation function 942a may include a logical component for assigning time information according to the role of each scene. This logical component may include provisions for optimally segmenting the overall timeline (e.g., 15 seconds) defined in the generation policy 200 according to the role of each scene (hook, development, benefit, close, etc.) and strictly assigning the start and end times of each scene. This allows for, for example, mathematically ensuring the overall rhythm of the video and that it fits within the specified time.

[0129] The storyboard generation function 942a may include a logical component relating to the structuring of visual direction. This logical component may include provisions for converting the placement and composition of subjects, camera work, lighting, and color design in each scene into specific linguistic instructions that can be interpreted by the AI ​​agent of the subsequent video generation function 942b. For example, the video generation function 942b generates specific video instructions that include the shooting intent, such as "macro photography (close-up) to highlight the texture of the product" or "medium shot to convey the liveliness of the store," based on a specific product image contained in the material information 120.

[0130] The storyboard generation function 942a may include a logical component for maintaining the identity of the subject. This logical component may include provisions that link the path of the source image to be referenced (image 122) and the features to be maintained (color, shape, texture) to the instructions for each scene, so that a specific object (e.g., a specific bowl of ramen, a logo, a person) appearing across multiple scenes does not change unnaturally between frames. This allows for the generation of constraints to minimize subject fluctuations specific to AI-generated videos, for example.

[0131] The storyboard generation function 942a may include a logical component relating to multimodal layer design. This logical component may include provisions for designing the position of text information to be superimposed on visual information, and the timing of sound effects to occur, to be synchronized with the movement of the video. This allows for the definition of time-axis event triggers within the storyboard 310, such as text appearing in accordance with a specific action in the video (e.g., the moment of draining water).

[0132] Thus, the storyboard generation function 942a, equipped with a visual design logic that includes time management, detailed video direction, and maintaining the identity of the subject, can output a storyboard 310 that serves as the basis for multimodal content 800 with a consistent storyline and brand quality, rather than simply a series of video materials.

[0133] In the example shown in Figure 5, the input information 100 includes an image 122 relating to the material of the multimodal content 800. In this example, the policy generation function 930 may generate a generation policy 200 that includes a policy prohibiting at least one attribute of the subject included in the image 122—namely, shape, color, components, logo, and price—from changing in a manner contrary to the laws of nature from the start to the end of the multimodal content 800. Here, changing in a manner contrary to the laws of nature may include objects unnaturally deforming, disappearing, or undergoing physically impossible color changes within the video. This makes it possible for a particular object to consistently maintain its shape, texture, and physical behavior from the start to the end of the video 320.

[0134] In the example shown in Figure 5, the AI ​​agent implementing the policy generation function 930 may incorporate policy generation guidelines to maintain the integrity of the image 122 included in the input information 100. Specifically, the policy generation function 930 generates a generation policy 200 for subsequent video generation functions 942b, etc., which includes instructions that strictly prohibit subject attributes from changing in a manner contrary to natural laws between the start and end points of the multimodal content 800.

[0135] In this example, an example of a system prompt input to the policy generation function 930 is outlined below. The system prompt may include the following: "System command: Formulate a dynamic consistency policy. Based on input information 100, output a generation policy 200 to maintain the immutability of the following attributes: 1. Maintaining shape and components: Prohibit the subject contained in image 122 (e.g., ramen ingredients, bowl, chopsticks) from being transformed into another object (e.g., bean sprouts changing into asparagus) in the middle of the video. 2. Color consistency: Prohibit the color of the subject defined in the material information (e.g., golden soup) from changing unnaturally beyond lighting effects. 3. Fixing logos and identification information: Prohibit the brand logo or text information contained in image 122 from being distorted or changing to a different design in the video. 4. Accuracy of price information: Prohibit the display of a numerical value in the video that differs from the price attribute specified in text data 124 (e.g., 1200 yen). Structure the above items as mandatory requirements for content policy 214."

[0136] The prompt description in the example shown in Figure 5 is just one example, and a person skilled in the art will recognize that if the description is conceptually similar, the same functions and constraints can be defined regardless of differences in detailed wording. The same applies to the subsequent examples of prompts.

[0137] In this way, the policy generation function 930 redefines what should not change based on the input information 100 from the perspective of natural laws and outputs it as a generation policy 200, so that subsequent video generation functions 942b etc. generate the video 320 under these constraints. This makes it possible to generate high-quality multimodal content 800 that does not cause discomfort to viewers while ensuring the credibility of the advertisement, for example.

[0138] In the example shown in Figure 5, the video generation function 942b may implement restrictive control that prohibits at least one attribute of the subject included in the image 122—namely, shape, color, components, logo, and price—from changing in a manner contrary to natural laws from the start to the end of the multimodal content 800. For example, the AI ​​agent implementing the video generation function 942b may have pre-built-in physical consistency prompts to ensure generation stability. Specifically, the AI ​​agent may contain a set of instructions that define the material persistence of an object and the interactions between objects. This makes it possible for a particular object to consistently maintain its shape, texture, and physical behavior from the start to the end of the video 320.

[0139] As a concrete example, when generating a cooking scene for "Homemade Noodles Shiroto," the agent for video generation function 942b may include prompts such as the following: "System Instructions: Strict adherence to dynamic consistency. 1. Material Persistence: Objects in the video (e.g., chopsticks and noodles) must maintain their shape between frames. Deformation into other objects is prohibited. 2. Gravity Consistency: All objects must obey gravity, and movement against gravity is prohibited (e.g., chopsticks not held in the hand must not float, water droplets when draining noodles must fall downwards in a parabolic arc). 3. Interaction: Simulate interactions between objects and prohibit non-physical passing through them (e.g., when chopsticks touch noodles, simulate the elasticity of the noodles and reproduce the behavior of noodles wrapping around the chopsticks)."

[0140] In this way, the video generation function 942b interprets the directorial intent specified in the storyboard 310 and applies rules based on natural laws incorporated as background knowledge, enabling the creation of multimodal content 800 with a realism equivalent to live-action footage that viewers will not find jarring. This allows for the maintenance of high credibility and brand image, for example, in advertising videos.

[0141] If the video generation function 942b is implemented by an AI agent, the AI ​​agent may include prompts that define video realization logic for generating high-quality video elements based on the storyboard 310 and source images, etc. These prompts may include the following logic components:

[0142] The video generation function 942b may include a logical component relating to the definition of physical causal relationships and interactions. This logical component may include provisions that strictly simulate causal relationships in which preceding actions lead to subsequent results for physical behavior occurring between objects in the video. This can, for example, define the fall of objects due to gravity or the change in texture resulting from the contact between cooking utensils and ingredients, thereby suppressing non-physical slippage and unnatural deformation.

[0143] The video generation function 942b may include a logical component relating to asset inheritance and the preferential use of edited images. This logical component may include a provision that prioritizes the use of specific image assets processed in a previous step as a source, rather than generating new video from scratch. This ensures that features that should not be altered, such as brand logos or product appearances, are inherited from the previous step, and maintains the identity of the subject throughout all frames of the video.

[0144] The video generation function 942b may include a logical component for motion planning based on scene attributes. This logical component may include provisions for calculating the optimal frame rate, motion speed, and subtle camera movements according to the role of the scene specified in the storyboard 310 (e.g., a scene emphasizing sizzle, a scene conveying vitality, etc.). This allows for control such as applying high-precision slow motion to scenes with splashing liquids and adding natural motion blur to scenes with moving people.

[0145] The video generation function 942b may include a logical component for sequential generation and state synchronization. This logical component may include provisions for sequentially executing each scene as an individual task, rather than generating multiple scenes at once, and reporting the completion status of each task to the command function 960 in real time. The save path of the generated video files may be managed using absolute paths, and metadata may be added to enable the subsequent editing function 950 to collect the materials without confusion.

[0146] Thus, the video generation function 942b, by incorporating a video realization logic that includes defining physical causal relationships, inheriting existing assets, and detailed motion control, can generate, for example, a video 320 that possesses the realism and appeal of live-action footage while maintaining the authenticity of the source material.

[0147] In the example shown in Figure 5, the BGM generation function 944a generates BGM 410 appropriate as background music for the content based on the generation policy 200 and at least one of the storyboard 310 and video 320. In this example, the BGM generation function 944a generates BGM 410 such as jazz or piano solo with a calm melody to match the store's concept of "redefining modern dining."

[0148] The element generation function 940 may classify the emotional elements to be included in the multimodal content 800 into multiple energy levels based on the generation policy 200, and control the composition of the audio information 400 based on the energy level. The element generation function 940 may control at least one of the volume, tempo, and beat intensity of the audio information 400, as well as the presence or absence of human voices. The composition of the audio information refers to elements used to express music, and may include, for example, the genre of music, the structure of the song, volume, tempo, time signature, beat intensity, instruments used, scenes, melody, harmony, presence or absence of singing, presence or absence of sound effects, etc. The energy level may be an index that expresses in stages the degree to which the multimodal content 800 can have an impact on the viewer's psychology, such as the heightened or intense emotions and the degree of tension.

[0149] In the example shown in Figure 5, the BGM generation function 944a may define the emotional elements to be included in the multimodal content 800 as multiple energy levels along a time axis, based on the generation policy 200 and at least one of the storyboard 310 and the video 320. The BGM generation function 944a may generate BGM 410 based on the defined energy levels. The BGM generation function 944a may control the intensity of the auditory impact of BGM 410 based on the defined energy levels. For example, the BGM generation function 944a may control the volume of the sound, tempo, number of instruments, frequency characteristics, and tension of the music based on the energy levels. If the defined energy levels are greater than a predetermined standard, the BGM generation function 944a may control the intensity of the auditory impact of BGM 410 to be greater than the corresponding standard.

[0150] The element generation function 940 may generate background music 410 included in audio information 400, based on the generation policy 200, under the constraint that it will be a single, consistent background music from the start to the end of the multimodal content 800.

[0151] In this example, the BGM generation function 944a extracts emotional parameters such as anticipation, appetite stimulation, calmness, and trust based on the generation policy 200, and converts these into energy levels ranging from level 1 to level 10. The BGM generation function 944a may dynamically change these energy levels in synchronization with the scene progression of the video. This makes it possible to generate BGM 410 that effectively influences the viewer's psychological state in accordance with the directorial intent. For example, in the opening scene of the video where the ramen first appears, the energy level is set low to stimulate the viewer's curiosity, and the background sound is controlled to be tranquil, such as a single piano note. Next, in a close-up scene where the yuzu espuma (foam) melts on top of the soup, the BGM generation function 944a raises the energy level to a medium level and controls the background sound to create anticipation for the change in taste by adding overlapping string instruments and sparkling high-frequency sounds. Next, in the scene where the cook drains the noodles, the BGM generation function 944a maximizes the energy level and creates a richer sound by emphasizing the rhythm and increasing the number of instruments, in order to express respect for the craftsman's skill. In the scene where customers are quietly enjoying their meal in the dining area, the BGM generation function 944a again lowers the energy level to create a calming, ambient background sound. The BGM generation function 944a may adjust the energy level from the start to the end of the multimodal content 800 so that the energy levels transition smoothly between multiple scenes, thereby generating a consistent BGM 410 overall.

[0152] In this way, by having the BGM generation function 944a control the energy level under the management of the command function 960, it is possible to generate highly immersive multimodal content 800 in which, for example, changes in visual information 300 such as video 320 and the emotional intensity of the BGM 410 are harmonized. This allows viewers to appreciate the high quality of the product and the brand value of the store on a deeper emotional level.

[0153] If the BGM generation function 944a is implemented by an AI agent, the AI ​​agent may include prompts that define sound design logic for generating music consistent throughout the video, based on the generation policy 200 and the storyboard 310. These prompts may include the following logic components:

[0154] The BGM generation function 944a may include a logical component for determining the sound policy based on emotional characteristics, without being limited to a specific music genre. This logical component may include provisions that define the core atmosphere of the music by verbalizing the emotional essence that the video should evoke (e.g., a sense of tranquil anticipation, a refined sense of dynamism, etc.), without relying on existing music genre names such as jazz or rock. This allows for, for example, the creation of original sound designs that match the brand's unique worldview.

[0155] The BGM generation function 944a may include a logical component relating to the hierarchical design of energy levels. This logical component may include provisions that define the build-up and tension of a piece of music as a multi-level numerical indicator and use this as a filter for the composition of the music. This allows, for example, the BGM generation function 944a to dynamically change the number of instruments used, the sound pressure, and the complexity of the rhythm based on these levels, thereby synchronizing the auditory impact with the scenes in the video.

[0156] The BGM generation function 944a may include a logical component relating to the definition of sound quality. This logical component may include not only melody and chord progressions, but also provisions that specifically define sound quality such as spatial breadth, resolution, bass depth, groove, and rhythm. This allows for the realization of the production policy 220 through an acoustic engineering approach, for example, by setting a deep reverberation in scenes that aim to create a sense of luxury, or increasing the clarity of the high frequencies in scenes that emphasize cleanliness.

[0157] The BGM generation function 944a may include a logical component relating to the development of a consistent motif. This logical component may include provisions for presenting a central short melody, motif, and rhythm at the beginning of the entire 15-second short content, and developing it in accordance with the progression of the scene, rather than simply repeating it. This ensures, for example, consistency as a single musical piece with a narrative from beginning to end, rather than being a combination of fragmented sound materials.

[0158] Thus, the BGM generation function 944a, equipped with acoustic design logic that includes the extraction of emotional essence, control by energy level, and definition of texture, can generate BGM 410 that, for example, resonates emotionally with visual information and maximizes the added value of the brand through auditory means.

[0159] In the example shown in Figure 5, the narration generation function 944b generates narration 420 based on the generation policy 200. For example, the narration generation function 944b is responsible for converting text data such as scripts included in the production policy 220 of the generation policy 200 into audio information using speech synthesis, and generating narration 420. The narration generation function 944b may generate narration 420 based on the generation policy 200, as well as at least one of the storyboard 310 and the animation 320. In this example, the narration generation function 944b may output a script describing the product's features, such as "a golden soup made with the rich flavors of whole Ooyama chicken and clams," as narration 420 in an elegant and calm tone that does not detract from the brand's sense of luxury and trustworthiness.

[0160] If the narration generation function 944b is implemented by an AI agent, the AI ​​agent may include prompts that define audio embodiment logic for converting the script specified in the generation policy 200 and storyboard 310 into high-quality audio information. These prompts may include the following logical components:

[0161] The narration generation function 944b may include a logical component to prevent alteration of the narration script. This logical component may include a provision that strictly prohibits the autonomous decision to modify, summarize, or embellish the narration script determined by the policy generation function 930. This prevents, for example, words that have been scrutinized according to intent information 110 from being unintentionally altered during the voice-to-narration stage, thereby ensuring the accuracy of the information.

[0162] The narration generation function 944b may include a logical component for determining speech characteristics according to the scene. This logical component may not simply read the text aloud, but may include provisions for selecting the optimal speech speed, pitch, and intonation pattern based on the purpose of the scene (hook, benefit, etc.) and a specified tone. This allows for control over selecting, for example, a clear and energetic tone in an opening scene to grab the viewer's attention, and a calm and stable tone in a scene that conveys trustworthiness.

[0163] The narration generation function 944b may include a logical component for separating visual and auditory information. This logical component may include a provision that clearly distinguishes between the text overlays and narration included in the storyboard 310 and limits the output to only the narration script. This eliminates interference such as the incorrect conversion of on-screen text into audio, and enables the generation of audio files that correctly reflect the division of information roles in multimodal content.

[0164] The narration generation function 944b may include a logical component for adapting the audio data to subsequent editing processes. This logical component assumes that the generated audio file will be combined with the background music 410 in the subsequent editing function 950, and may include provisions for maintaining clear, noise-free sound quality and file management using a specified save format and absolute path. This logical component may include provisions for checking whether the narration for each scene fits within the time slot defined in the storyboard 310, and making adjustments as necessary.

[0165] Thus, the narration generation function 944b, by incorporating features such as preventing alteration of the narration script, determining speech characteristics according to the scene, and logic for adapting the audio data to subsequent editing processes, can generate narration 420 that, for example, harmonizes well with visual information and background music, and delivers the brand message accurately and attractively to the listener's ears.

[0166] In the example shown in Figure 5, the text information generation function 946 generates text information 500 based on the generation policy 200. The text information 500 may include captions and subtitles superimposed on the video. For example, the text information generation function 946 generates captions such as "Is this ramen...?" or "A bowl of ramen that your body will love, completely free of artificial additives" based on the content policy 214.

[0167] If the text information generation function 946 is implemented by an AI agent, the AI ​​agent may include prompts that define text design logic for generating text information to be superimposed on the video based on the generation policy 200. These prompts may include the following logical components:

[0168] The text information generation function 946 may include a logical component for adjusting character placement and display methods, prioritizing format policies. This logical component may include provisions for adjusting the physical format without compromising the meaning of the content, while strictly adhering to the character and line limits specified in format policy 212. For example, if important persuasive language exceeds the character limit for a single line, the authenticity of the information can be maintained by prioritizing line breaks or splitting the text into multiple parts, rather than autonomously summarizing or deleting the phrase.

[0169] The text information generation function 946 may include a logical component for determining typographic attributes. This logical component may include provisions that define the font type, size, color, and border style that match the tone specified in the presentation policy 220 (e.g., sophisticated, approachable). This logical component may also include provisions that dynamically specify text decorations (such as outlines and drop shadows) to maximize readability, taking into account the color distribution of the background video 320.

[0170] The text information generation function 946 may include a logical component for controlling the appearance and disappearance sequence. This logical component may include provisions for defining the timing of when the text appears and disappears on the screen, synchronized with the time slots of each scene defined in the storyboard 310. It may also include provisions for selecting not only simple display but also types of text animation, such as fade-in, slide, or typing-style, according to the energy level of the scene.

[0171] The text information generation function 946 may include a logical component for verifying multilingual and regional suitability. This logical component may include provisions for complying with notation rules and prohibited vocabulary specific to a specified output language. The text information generation function 946 may also include provisions for self-verifying whether expressions or symbols considered inappropriate in a particular region are included, and, if necessary, making minor adjustments to culturally appropriate expressions while maintaining meaning.

[0172] Thus, by incorporating formal constraint compliance, typography design, and text appearance timing design logic including time axis synchronization, the text information generation function 946 can generate text information 500 that effectively complements the brand message as text information without hindering the appeal of the video as visual information.

[0173] The element generation function 940 may include the format policy 212 from the generation policy 200 in the product generated by the generation of at least two of the information from visual information 300, audio information 400, and text information 500. For example, when the visual information generation function 942 generates visual information 300 and then the audio information generation function 944 generates audio information 400, the visual information generation function 942 includes the format policy 212 in the generated visual information 300. As a result, the audio information generation function 944, which is responsible for the next step, will generate the audio information 400 based on the format policy 212 included in the visual information 300 in addition to the format policy 212 included in the generation policy 200, so that the audio information 400 can be generated in compliance with the format policy 212. In this way, by including the format policy 212 in the intermediate product generated by one function, the likelihood of other functions responsible for the next step complying with the format policy 212 can be increased. The element generation function 940 may also include the format policy 212 and the content policy 214 in the intermediate product. The element generation function 940 may include the entire generation policy 200 as an intermediate product.

[0174] In the example shown in Figure 5, the input information 100 may include reference content 130. The policy generation function 930 may generate a generated policy 200 based on the reference content 130. The element generation function 940 may generate at least two of the following information: visual information 300, audio information 400, and text information 500, based on the reference content 130. The reference content 130 may be information illustrating a completed image of the multimodal content 800. The reference content 130 may include, for example, existing video files, audio files, and still image files.

[0175] In the example shown in Figure 5, the policy generation function 930 may extract features such as color composition, composition, editing rhythm, BGM tone, and production techniques from the reference content 130. The policy generation function 930 may reflect the extracted features in the generated policy 200. For example, the policy generation function 930 may reflect the extracted features in the production policy 220. The policy generation function 930 may obtain these features by having the analysis function 920 analyze the reference content 130. This allows the system to autonomously design multimodal content 800 with a worldview that matches the user's preferences, even if the user cannot verbalize specific instructions, simply by presenting the reference content 130.

[0176] Let's explain using the example of "Homemade Noodles Shiroto". For example, as reference content 130, a video introducing a high-end restaurant that makes extensive use of cinematic lighting and slow motion is input. In this case, the policy generation function 930 extracts the texture of the production, such as "low-light, high-contrast lighting" and "slow-motion footage with a high frame rate," from the video.

[0177] Based on these extraction results, the policy generation function 930 generates a production policy 200 that includes specific instructions for the video generation function 942b, such as "compose the entire video with a calm, low-saturation tone" as the production policy 220, and "generate the draining and pouring scenes in slow motion." In this way, by using the reference content 130, the newly generated video 320 and BGM 410 can inherit the unique atmosphere of a particular brand or genre.

[0178] In the example shown in Figure 5, the reference content 130 may include video data, still image data, and audio data shot by the user themselves. In this example, the information processing system 90 may not only use the reference content 130 as a mere source for direction, but may also incorporate the content itself as primary material and perform processing to refine its expression to professional quality.

[0179] In the example shown in Figure 5, for instance, the owner of "Homemade Noodles Shiroto" films himself draining noodles in the kitchen using a simple camera and inputs this video as reference content 130 into the information processing system 90. The visual information generation function 942 maintains the composition and movement of the subject in the input video and may use AI to optimize lighting, remove noise, create smooth slow motion through frame interpolation, and apply color design such as color grading. This allows the video, even if not shot by a professional cameraman, to be transformed into a cinematic video that conforms to the production policy 220 of the generation policy 200.

[0180] In the example shown in Figure 5, the audio information generation function 944 may extract ambient sounds such as the sound of noodles being pounded and the sound of soup being poured from the reference content 130, and adjust them to more appetizing and emphasized sound effects based on the production policy 220. In this way, the information processing system 90 can generate high-quality multimodal content 800 that enhances brand value while making the most of the realism of the primary material by supplementing and strengthening any missing production elements based on the reference content 130 provided by the user. This allows even users without specialized filming or editing skills to obtain professional advertising videos and the like using their own materials.

[0181] In the example shown in Figure 5, the change management function 970 may manage whether the input information 100 and the intermediate products generated by each function can be changed. For example, the change management function 970 may manage whether the input information 100, the generation policy 200, the visual information 300, the audio information 400, and the text information 500 can be changed. The change management function 970 may manage whether the storyboard 310, the video 320, the background music 410, and the narration 420 can be changed. The change management function 970 may also manage whether the multimodal content 800 can be changed.

[0182] In the example shown in Figure 5, the change management function 970 may lock information generated by a function by setting it to an unmodifiable state once it determines that the information has been finalized. In this example, the change management function 970 controls the function executing subsequent processes to prohibit modification of the locked information. This prevents important intentions and constraints determined in previous processes from being inadvertently overwritten by subsequent processes during the autonomous collaboration of multiple AI agents, thereby ensuring the consistency of the entire content.

[0183] In the example shown in Figure 5, the change management function 970 may change the status of the information from changeable to unchangeable, triggered by an approval operation from the user or passing a quality check by the command function 960.

[0184] In the example shown in Figure 5, for example, the policy generation function 930 generates a generation policy 200 that includes a content policy 214 stating "To create a sense of luxury, the notation 'free extra servings' is prohibited," saves it to the storage device 909, and locks it with the change management function 970. If the subsequent text information generation function 946, based on general, statistical ramen advertisement learning data, were to try to generate a caption suggestion of "free extra servings" with good intentions, the change management function 970 would reject the change or addition because it has locked the referenced policy. Similarly, after the storyboard 310 is finalized and locked, the video generation function 942b cannot arbitrarily change the composition specified in the storyboard 310, which is "close-up of yuzu espuma (foam)."

[0185] In this way, the change management function 970 strictly manages the status of intermediate products, structurally preventing deviations from the intended design due to the uncertainties inherent in large-scale language models and generative AI. This allows for the generation of the final multimodal content 800, faithfully reflecting the concept defined in the input information 100.

[0186] If the change management function 970 is implemented by an AI agent, the AI ​​agent may include workflow consistency logic to monitor the finalization status of intermediate products by each specialist agent and prevent unintentional modifications or deviations from the process. This logic may include the following logical components:

[0187] The change management function 970 may include a logical component for switching the processing progress status according to the completion status of each sub-process. This logical component may define each sub-process of content generation as an independent stage and include a provision that fixes the deliverables and makes them unchangeable when each stage transitions to a completed state. This structurally suppresses inconsistencies between processes, such as retroactively rewriting the content of an already finalized scenario in the audio generation process.

[0188] The change management function 970 may include a logical component for centrally managing the storage location and related information of each deliverable. This logical component may include provisions for centrally managing the absolute paths and metadata of intermediate products generated by each agent as a shared workflow state. For example, each agent can refer to the centrally managed workflow state information before autonomously starting processing to confirm that the deliverables of the preceding process are correctly registered. This can maintain an environment where, for example, only the latest and finalized materials are inherited by subsequent processes.

[0189] The change management function 970 may include a logical component for determining whether the next process can be started based on confirmation of the completion of the preceding process. This logical component may include a provision that enforces the completion of a prerequisite preceding subprocess as a condition for starting the execution of a specific subprocess. For example, this logical component may prohibit the start of the storyboard generation process or the animation generation process at the system level if the policy generation process is not completed. This can prevent, for example, the wasteful consumption of resources in the absence of blueprints or the generation of inconsistent materials.

[0190] The change management function 970 may include logical components related to traceability of execution history and error recovery. These logical components may include provisions that record execution status reports from each agent along with timestamps, and enable root cause analysis when an anomaly occurs, determining which agent failed, what material it used, and why. Furthermore, when a failure is reported, these logical components may include provisions that, in cooperation with the command function 960, provide information to determine whether to safely retry or terminate the process.

[0191] Thus, the change management function 970 incorporates workflow consistency logic, including stage-specific state locking, centralized management of deliverables, verification of process dependencies, and recording of execution history. This allows for asynchronous and parallel processing by multiple AI agents, enabling, for example, the process to reach the final multimodal content 800 without deviating from the original intent.

[0192] In the example shown in Figure 5, the quality check function 980 may evaluate the product generated by the policy generation function 930 based on the input information 100. The quality check function 980 may evaluate the product generated by the element generation function 940 based on at least one of the input information 100 and the generation policy 200. The quality check function 980 may evaluate the product generated by the editing function 950 based on at least one of the input information 100 and the generation policy 200. The quality check function 980 may evaluate not only the final product from each function, but also the products at intermediate stages of generation. For example, the quality check function 980 may evaluate not only the generation policy 200 that the policy generation function 930 ultimately generates, but also the products at intermediate stages in the generation of the generation policy 200. The quality check function 980 may output these evaluation results as content evaluation results. The quality check function 980 may feed back the evaluation results to the function that generated the product. The quality check function 980 may feed back the evaluation results to the command function 960. The command function 960 may adjust the execution status of the generation of multimodal content 800 based on the content evaluation results.

[0193] In the example shown in Figure 5, the quality check function 980 may perform content evaluation on the generated product based on multifaceted indicators. The quality check function 980 may perform content evaluation on the generated intermediate product and the final multimodal content 800 based on multifaceted indicators. The quality check function 980 may achieve quality improvement not only by making pass / fail judgments based on a single criterion, but also by performing evaluations based on multiple variations.

[0194] The quality check function 980 may perform a logical consistency evaluation regarding the generation policy 200. The quality check function 980 may determine whether the product under evaluation contains all the required items specified in the content policy 214, or whether any prohibited items are included. In the example of "Homemade Noodles Shiro to", the quality check function 980 analyzes the text overlays and narration 420 in the video 320 to check for the presence of prohibited keywords such as "large portion" and "hearty". The quality check function 980 may also check whether the price information is displayed correctly as "1200 yen".

[0195] The quality check function 980 may perform a physical validity assessment. The quality check function 980 may determine whether there are any unnatural depictions caused by the generating AI. For example, the quality check function 980 may determine whether the number of chopsticks in the video 320 generated by the video generation function 942b has increased or decreased unnaturally, whether the texture of the noodles has collapsed like a liquid, or whether the background store logo has distorted between frames. The quality check function 980 may perform this determination using an image analysis agent. If an unnatural part is detected, the quality check function 980 may output specific error information, such as "Physical inconsistency: Deformed chopstick shape," to the command function 960 and prompt corrective action such as re-execution.

[0196] The quality check function 980 may perform sensory and emotional evaluations based on the production policy 220. The quality check function 980 may quantify whether the target product matches the intended tone and manner. For example, the quality check function 980 may score abstract attributes such as a sense of luxury and cleanliness based on color distribution, BGM tempo, and editing rhythm. In the example of "Homemade Noodles Shiro to," the quality check function 980 evaluates whether the tone of the video is too dark and whether the BGM 410 is too noisy to disrupt the "tranquil atmosphere" of the store.

[0197] The quality check function 980 may perform an effectiveness evaluation. For example, the effectiveness evaluation may be a marketing predictive evaluation that predicts the response of the target customer group. The quality check function 980 may use an AI model to predict what click-through rate and viewer retention rate the generated content will achieve based on past advertising operation data, etc. The quality check function 980 may compare multiple variations of the generated product, select the product that is expected to have the highest effect, and reject the others. The quality check function 980 may perform A / B testing.

[0198] The quality check function 980 may perform a multifaceted evaluation from four perspectives: logical consistency evaluation, physical validity evaluation, sensory and emotional evaluation, and effectiveness evaluation. The quality check function 980 may perform an evaluation that combines at least two of these evaluations. For example, if each of the two evaluation results meets a predetermined standard, it may be judged as passing, and if either evaluation result does not meet the standard, it may be judged as failing. This allows, for example, the information processing system 90 to stably output high-quality multimodal content 800 suitable for commercial use while minimizing supervision by a human director.

[0199] If the quality check function 980 is implemented by an AI agent, the AI ​​agent may include quality control logic for verifying whether the intermediate products and the final multimodal content 800 conform to the generation policy 200 and physical validity. This logic may include the following logical components. The quality check function 980 may also be implemented as self-assessment logic implemented within each specialist agent, in which case each specialist agent may include the following logical components.

[0200] The quality check function 980 may include a logical component for verifying the consistency of the thinking process. This logical component may include provisions to verify whether each expert agent has correctly followed all the thinking steps defined by the prompt, such as material analysis, problem identification, and solution formulation, before executing any tools. If any omissions or inconsistencies are detected in the thinking process, provisions may be included to suspend the execution of the tools and force a retry of the thinking process. This can, for example, prevent the occurrence of unfounded hallucinations.

[0201] The quality check function 980 may include a logical component for evaluating compliance with absolute constraints on deliverables. This logical component may include provisions for determining whether the generated text or video satisfies formal and substantive constraints. For example, it may include provisions for verifying, through rigorous pattern matching and semantic analysis, whether unauthorized alterations have been made to the script during narration generation, or whether the character limit has been exceeded during text generation.

[0202] The quality check function 980 may include a logical component for verifying multimodal temporal consistency. This logical component may include provisions for verifying whether the visual information 300, audio information 400, and text information 500 are arranged in a manner consistent with the timeline defined in the storyboard 310. For example, it may include provisions for checking, on a frame-by-frame and millisecond-by-millisecond basis, whether the length of the narration exceeds the allocated scene time, or whether the timing of the appearance of the captions is synchronized with the transitions in the video.

[0203] The quality check function 980 may include a logical component relating to the censorship of physical laws and causal relationships. This logical component may include provisions for evaluating, through image analysis, whether the behavior of objects in the generated video 320 violates natural laws. It may also include provisions for verifying the validity of error logs during tool execution and the absolute paths of generated files, and for making a final determination of systemic feasibility.

[0204] Thus, the quality check function 980 incorporates quality control logic that includes verification of thought processes, assessment of the suitability of absolute constraints, confirmation of time axis synchronization, and censorship of physical validity. This allows for the stable output of multimodal content 800 that maintains reliability and completeness suitable for commercial use, even without detailed supervision by a human director.

[0205] In the example shown in Figure 5, the adjustment of the execution status of the generation of multimodal content 800 by the command function 960 may include at least one of the following: modification, addition, and deletion of processes. The adjustment of the execution status may include instructing a retry of the process, branching the generation process, changing the generation method, and adjusting the parameters of subsequent steps. The adjustment of the execution status may include adding and executing a new process, returning and re-executing some or all of the processes, and changing some or all of the execution order of the processes and re-executing some or all of them.

[0206] In the example shown in Figure 5, if the evaluation result from the quality check function 980 does not meet predetermined criteria, the command function 960 may share the specific error information, improvement feedback, etc. included in the evaluation result with the function that generated the relevant product and instruct it to re-execute. For example, if a deformation of the chopsticks shape is detected in the physical validity evaluation, the command function 960 will dynamically adjust the prompt to the video generation function 942b to correct the erroneous part and then instruct it to regenerate the video 320. Alternatively, the command function 960 may issue only a re-execution instruction without specific instructions, and the video generation function 942b may correct the relevant part of the video 320 based on the error information shared by the command function 960.

[0207] In the example shown in Figure 5, the command function 960 may dynamically adjust the models and resources used in subsequent processes based on the content evaluation results. For example, if, in the affective evaluation, the "luxury" score of the generated video 320 reaches the target value but is within an acceptable range, the command function 960 may give additional instructions to the subsequent BGM generation function 944a, such as "make a more substantial cello solo the main element" to compensate for the lack of luxury, and have it execute. This enables the optimization of quality through mutual complementarity between processes.

[0208] In the example shown in Figure 5, the command function 960 may, based on the results of the effectiveness evaluation, select the optimal one from among multiple generation variations and make adjustments to terminate the remaining processing. For example, if the editing function 950 generates multiple A / B test options, the command function 960 may adopt only the option with the highest predicted click-through rate by the quality check function 980 as the final multimodal content 800 and pass it to the output control function 990.

[0209] By having the command function 960 reflect the evaluation results in the execution status in real time, the information processing system 90, for example, can achieve early detection and correction of errors, and reach the target quality via the shortest path.

[0210] In the example shown in Figure 5, the storage device 909 may be a data storage device that is commonly accessible to all functions within the information processing system 90. The storage device 909 may store input information 100, generation policies 200, and intermediate products generated in each process, along with version information representing the history of the generation process. In this example, the change management function 970 may control whether or not changes can be made to each data stored in the storage device 909 by performing flag management, etc. The storage device 909 may store multimodal content 800 generated in the past and evaluation results from the quality check function 980 for that content. For example, by having the AI ​​agent implementing each function refer to the data of past adopted and rejected products stored in the storage device 909 and the reasons for their rejection, a learning effect can be obtained that improves the accuracy of generation with each iteration.

[0211] In the example shown in Figure 5, the output control function 990 may be controlled to output various types of information. For example, the output control function 990 may be controlled to output multimodal content 800. The information output by the output control function 990 may include transmission output and display output. The information output by the output control function 990 may also include granting access rights to the target information. For example, the output control function 990 may grant access rights to the target information stored in the storage device 909 to predetermined access sources.

[0212] In the example shown in Figure 5, the output control function 990 may output the multimodal content 800 with tracking information added for rights protection and traceability. For example, the output control function 990 may add a digital watermark to the multimodal content 800. The output control function 990 may embed invisible or inaudible information that is difficult to discern with the naked eye or ear into the video and audio of the multimodal content 800. This makes it possible to identify the source of unauthorized distribution of the content and to add identification information indicating that the content was generated by AI, thereby protecting brand rights and ensuring transparency of information.

[0213] The output control function 990 may output the multimodal content 800 after performing format conversion to a format suitable for the output destination, resolution adjustment, bitrate optimization, etc. The output control function 990 may control the output of the multimodal content 800 in a form appropriate to the distribution medium. For example, the output control function 990 may adjust the output of multimodal content 800 that has been simply adjusted so that multimodal content 800 generated on the premise of being viewed on a vertical display can also be viewed on a horizontal display. The output control function 990 may also control the output of multimodal content 800 with some elements removed from it, together with the original multimodal content 800. For example, if the original multimodal content 800 includes a caption, the output control function 990 may control the output of the lower multimodal content 800 with the caption and the multimodal content 800 without the caption together.

[0214] If the output control function 990 is implemented by an AI agent, the AI ​​agent may include output management logic to present the generated multimodal content 800 in a user-available state and to safely shut down the operation of the entire system. This logic may include the following logical components:

[0215] The output control function 990 may include a logical component for determining and notifying the storage location of the final deliverable. This logical component may include provisions for verifying that the final multimodal content 800 output by the editing function 950, etc., exists in the correct absolute path on the storage and for accurately communicating this to the user interface or subsequent distribution system. The file name may also include provisions for adding metadata such as the tenant name, product name, and timestamp to ensure ease of management.

[0216] The output control function 990 may include a logical component for confirming and notifying completion of processing. This logical component may include a provision for issuing a termination command to the entire system indicating the completion of all processes immediately after confirming that all sub-processes are complete and the final deliverable has been output. This allows, for example, each autonomously operating agent to be explicitly instructed to complete the project and release computing resources.

[0217] The output control function 990 may include a logical component for organizing multiple output variations so that they can be output together. This logical component may include provisions for collecting not only a single completed video, but also all variations required by the generation policy 200 (e.g., versions with and without subtitles, versions with different aspect ratios, etc.), and providing them to the user all at once. This can, for example, support the optimal distribution of materials to match the specifications of each advertising distribution medium.

[0218] The output control function 990 may include a logical component for recovery reporting in the event of abnormal termination. This logical component may include a provision for terminating a sub-process if it is determined that output at the target quality is difficult due to a failure or resource depletion in a specific sub-process, along with the status of the intermediate products up to that point and the specific reasons why termination was unavoidable. This allows, for example, the user to be provided with specific improvement information for the next trial without receiving an unclear error message.

[0219] Thus, the output control function 990, by incorporating output management logic including the storage location of the final output, confirmation of processing completion, variation management, and closing processing in case of abnormalities, can reliably complete the complex multi-agent generation process and realize the provision of highly reliable multimodal content 800.

[0220] In the example shown in Figure 5, the multimodal content 800 may be video content including visual information 300, audio information 400, and text information 500. In this example, the element generation function 940 may have a storyboard generation function 942a, a video generation function 942b, a background music generation function 944a, and a narration generation function 944b. In this example, the input information 100 may include intent information 110 representing the intention to produce the video content and material information 120 relating to the materials of the video content. In this example, the command function 960 may manage the start and completion of execution of multiple functions provided by the information processing system 90. In this example, the policy generation function 930 may generate a generation policy 200 based on the intent information 110 and material information 120 in response to the execution start process by the command function 960. In this example, the command function 960 may, upon completion of the execution of the policy generation function 930, determine the execution order of multiple functions, including the storyboard generation function 942a, the video generation function 942b, the background music generation function 944a, and the narration generation function 944b, based on the generated policy 200 generated by the policy generation function 930, and manage to sequentially execute the process of starting one of the multiple functions according to the determined execution order, and starting the next function in the determined execution order upon completion of the execution of that one function.

[0221] This allows the policy generation function 930 to pre-formulate a generation policy 200 based on the production intent and material information, and each specialized function to follow this policy. For example, this prevents discrepancies in tone and content between visual, audio, and text elements, enabling the generation of high-quality, consistent video content. The command function 960 dynamically determines the optimal execution order according to the content of the generation policy 200 and manages the start and completion of each function. This reduces the effort required for human direction and process management, resulting in shorter production times and reduced costs. The information processing system 90 determines the optimal order based on the materials and intent, such as "first finalize the storyboard, and then generate the video and narration," enabling the generation of content in the most optimal process for a wide variety of materials and production needs.

[0222] In the example shown in Figure 5, the input information 100 may include intent information 110 that represents the production intent of the multimodal content 800. In this example, the policy generation function 930 may convert the information expressed from the perspective of the creator of the multimodal content 800 into information expressed from the perspective of the viewer of the multimodal content 800. In this example, the policy generation function 930 may generate the generated policy 200 based on the information expressed from the viewer's perspective. This allows the policy generation function 930 to convert the creator's expertise and often self-righteous preferences entered by the user on the production side into customer value that aligns with the viewer's interests and experience, thereby generating highly empathetic content that matches the viewer's needs. Because the generated policy 200 can be formulated after reconstructing the creator's technical terms and explanations into expressions that are easy for viewers to understand and appealing, it becomes possible to communicate the value of the advertised brand correctly and effectively without damaging its brand image. By incorporating a market-in perspective—how it looks—instead of a product-out approach focused on what message to convey, the information processing system 90 enables even users without advanced marketing knowledge to acquire highly competitive multimodal content 800, similar to that created by a professional creative director.

[0223] In the example shown in Figure 5, the information processing system 90 may have multiple AI agents. In this example, multiple AI agents may realize multiple functions as shown in Figure 5. In this example, the AI ​​agent that implements the command function 960 and the AI ​​agent that implements the element generation function 940 may be different from the other AI agents. Among the multiple AI agents, the AI ​​agent that implements the command function 960, the AI ​​agent that implements the policy generation function 930, and the AI ​​agent that implements the element generation function 940 may be different. Among the multiple AI agents, the AI ​​agent that implements the command function 960, the AI ​​agent that implements the policy generation function 930, the AI ​​agent that implements the element generation function 940, and the AI ​​agent that implements the quality check function 980 may be different.

[0224] In this embodiment, different AI agents may include the AI ​​agents running as logically independent processes. Different AI agents may include the AI ​​agents being defined by prompts with different content. Different AI agents may include the AI ​​agents operating by different models. Different AI agents may include the AI ​​agents operating in independent memory spaces.

[0225] By having different AI agents specializing in different areas handle management tasks such as command and creative tasks such as generation, more advanced and specialized processing becomes possible at each stage compared to, for example, having a single AI model perform everything, thereby improving the quality of the final multimodal content 800. Logically separating the command function 960 and the element generation function 940 structurally suppresses, for example, the fabrication of information and deviations from the production intent. Since each function is implemented as an independent AI agent, even if there is a technological innovation in a particular media, flexible upgrades can be performed by replacing only the relevant agent with the latest model without having to rebuild the entire system. Because the command function acts as an independent command center, it becomes easy to run multiple element generation agents (visual, audio, text, etc.) simultaneously in parallel, and in some cases, multimodal content 800 can be produced in a shorter time compared to a production flow where humans give sequential instructions.

[0226] In the example shown in Figure 5, the policy generation function 930, element generation function 940, editing function 950, and command function 960 are each implemented by an AI agent, and the AI ​​agents that implement each of the policy generation function 930, element generation function 940, editing function 950, and command function 960 may be different from each other. In this example, the AI ​​agents that implement each of the functions included in the element generation function 940 may be different from each other. The AI ​​agents that implement the storyboard generation function 942a, video generation function 942b, BGM generation function 944a, narration generation function 944b, and text information generation function 946 may be different from each other. These AI agents may be different from the AI ​​agents that implement each of the policy generation function 930, editing function 950, and command function 960.

[0227] Figure 6 schematically shows an example of processing by the information processing system 90. In the example shown in Figure 6, the process from when the information processing system 90 obtains input information 100 from the user 10 to when it outputs multimodal content 800 to the user 10 is described. In this example, the command function 960 determines whether or not each sub-process performed by the functions of the information processing system 90 has been completed, and if it has been completed, it issues an instruction to start the next sub-process, and if it has not been completed, it does not issue an instruction to start the next sub-process.

[0228] In step 102 (sometimes abbreviated as S), the information processing device 900 receives the input information 100.

[0229] In S104, based on the input information 100 received by the information processing device 900 in S102, the command function 960 instructs the policy generation function 930 to generate the generation policy 200. The command function 960 may schedule the generation of the multimodal content 800 in response to the information processing device 900 receiving the input information 100, and may give subsequent instructions based on the scheduled process chart.

[0230] In S106, the policy generation function 930 generates a generated policy 200 based on the input information 100 received by the information processing device 900 in S102, in response to instructions received from the command function 960 in S104, and sends the generated policy 200 to the information processing device 900.

[0231] In S108, the command function 960 determines whether the generation of the generated policy 200 by the policy generation function 930 has been completed. If the command function 960 determines that the generation process is complete, it proceeds to the next step; if it determines that it is not complete, it does not proceed to the next step. For example, if the generation is not completed within a predetermined period, the command function 960 outputs information to the user 10 indicating a timeout error and terminates the process prematurely.

[0232] In S110, the command function 960 instructs the storyboard generation function 942a to generate storyboard 310.

[0233] In S112, based on the input information 100 received by the information processing device 900 in S102 and the generation policy 200 generated by the policy generation function 930 in S106, the storyboard generation function 942a generates a storyboard 310 and sends the storyboard 310 to the information processing device 900.

[0234] In S114, the command function 960 determines whether the generation of the storyboard 310 by the storyboard generation function 942a has been completed. If the command function 960 determines that the generation process is complete, it proceeds to the next step; if it determines that it is not complete, it does not proceed to the next step. For example, if the generation is not completed within a predetermined period, the command function 960 outputs information to the user 10 indicating a timeout error and terminates the process prematurely.

[0235] In S116, the command function 960 instructs the video generation function 942b to generate the video 320.

[0236] In S118, based on the input information 100 received by the information processing device 900 in S102 and the generation policy 200 generated by the policy generation function 930 in S106, the video generation function 942b generates a video 320 and transmits the video 320 to the information processing device 900. The video generation function 942b may further generate a video 320 in accordance with the storyboard 310 generated by the storyboard generation function 942a in S112.

[0237] In S120, the command function 960 determines whether the video generation function 942b has completed generating the video 320. If the command function 960 determines that the generation process is complete, it proceeds to the next step; if it determines that it is not complete, it does not proceed to the next step. For example, if the generation is not completed within a predetermined period, the command function 960 outputs information to the user 10 indicating a timeout error and terminates the process prematurely.

[0238] In S122, the command function 960 instructs the BGM generation function 944a to generate BGM 410.

[0239] In S124, based on the input information 100 received by the information processing device 900 in S102 and the generated policy 200 generated by the policy generation function 930 in S106, the BGM generation function 944a generates BGM 410 and transmits BGM 410 to the information processing device 900. The BGM generation function 944a may further generate BGM 410 in accordance with the storyboard 310 generated by the storyboard generation function 942a in S112. The BGM generation function 944a may further generate BGM 410 in accordance with the video 320 generated by the video generation function 942b in S118.

[0240] In S126, the command function 960 determines whether the generation of BGM 410 by the BGM generation function 944a has been completed. If the command function 960 determines that the generation process is complete, it proceeds to the next step; if it determines that it is not complete, it does not proceed to the next step. For example, if the generation is not completed within a predetermined period, the command function 960 outputs information to the user 10 indicating a timeout error and terminates the process prematurely.

[0241] In S128, the command function 960 instructs the editing function 950 to generate the multimodal content 800.

[0242] In S130, the editing function 950 generates multimodal content 800 by editing the video 320 and BGM 410. The editing function 950 may generate multimodal content 800 by editing the video 320 and BGM 410 based on the input information 100 received by the information processing device 900 in S102 and the generation policy 200 generated by the policy generation function 930 in S106. The editing function 950 may generate multimodal content 800 by editing the video 320 and BGM 410 based on the generation policy 200 and the storyboard 310. For example, the editing function 950 superimposes BGM 410 onto the video 320 so as to satisfy the constraints 210 of the generation policy 200 while achieving the production policy 220. The editing function 950 transmits the generated multimodal content 800 to the information processing device 900.

[0243] In S132, the command function 960 determines whether the generation of the multimodal content 800 by the editing function 950 has been completed. If the command function 960 determines that the generation process is complete, it proceeds to the next step; if it determines that it is not complete, it does not proceed to the next step. For example, if the generation is not completed within a predetermined period, the command function 960 outputs information to the user 10 indicating a timeout error and terminates the process prematurely.

[0244] In S134, the command function 960 operates the information processing device 900 to output the multimodal content 800 generated by the editing function 950 in S130 to the user. If the information processing device 900 is equipped with an output control function 990, the command function 960 may instruct the output control function 990 to output the multimodal content 800 generated by the editing function 950 to the user, and the output control function 990 may output the multimodal content 800 in accordance with that instruction. The output of the multimodal content 800 may be a display output to a display device equipped with the information processing device 900. The output of the multimodal content 800 may be a transmission output to the user 10's local terminal.

[0245] In the example shown in Figure 6, the process is described as being executed sequentially in the order of generating the generation policy 200 in S106, then generating the storyboard 310, then the animation 320, and then the background music 410. However, the order of processing after the generation policy 200 is not limited to this. For example, these may be executed in a different order. For example, after the generation policy 200 in S106, the process may be executed sequentially in the order of generating the storyboard 310, then the background music 410, and then the animation 320. The processing after the generation policy 200 is executed may include parallel processing. For example, after the generation policy 200 in S106 is generated, the storyboard 310 may be generated, and then the animation 320 and background music 410 may be generated in parallel.

[0246] Figure 7 schematically shows an example of the information processing system 90. This example mainly explains the differences from the examples shown in Figures 3 and 4. This example shows an example of the fully localized implementation described above.

[0247] In the example shown in Figure 7, the information processing system 90 is integrated into a single device. In this example, the information processing system 90 is configured as an information processing device 900. In this example, all of the multiple functions of the information processing system 90 are implemented in the information processing device 900. In this example, each of the multiple functions is connected via a communication channel 97 for communication.

[0248] In the example shown in Figure 7, if each of the multiple functions is an AI, each function may utilize an external API that is connected to the information processing device 900 in a communicative manner. Each function may also operate in a standalone environment within the information processing device 900. Some of the functions may utilize external APIs, etc., while the remaining parts operate in a standalone environment within the information processing device 900. The AI ​​may be an AI agent.

[0249] Figure 8 schematically shows an example of the processing flow by the information processing system 90. In the example shown in Figure 8, we will describe the case where the information processing system 90 is equipped with a policy generation function 930, a storyboard generation function 942a, a video generation function 942b, a background music generation function 944a, an editing function 950, and a command function 960.

[0250] In the example shown in Figure 8, the processing flow from when the information processing system 90 acquires input information 100 until it completes the generation of multimodal content 800 will be explained. For brevity, the details are omitted from Figure 8, but in the example shown in Figure 8, as in the example shown in Figure 6, the command function 960 determines whether or not the sub-process performed by each function of the information processing system 90 has been completed. If it has been completed, it issues an instruction to start the next sub-process; if it has not been completed, it does not issue an instruction to start the next sub-process.

[0251] In S202, the command function 960 receives input information 100.

[0252] In S204, the command function 960 instructs the policy generation function 930 to generate a generated policy 200, and the policy generation function 930 generates the generated policy 200 based on this instruction and the input information 100 received by the command function 960 in S202.

[0253] In S206, upon completion of the generation of the generation policy 200 by the policy generation function 930, the command function 960 instructs the storyboard generation function 942a to generate the storyboard 310, and the storyboard generation function 942a generates the storyboard 310 based on this instruction, the input information 100, and the generation policy 200 generated by the policy generation function 930 in S204.

[0254] In S208, upon completion of the storyboard generation function 942a's generation of storyboard 310, the command function 960 instructs the animation generation function 942b to generate animation 320. The animation generation function 942b then generates animation 320 based on this instruction, input information 100, generation policy 200, and the storyboard 310 generated by the storyboard generation function 942a in S206.

[0255] In S210, upon completion of video generation 320 by video generation function 942b, the command function 960 instructs BGM generation function 944a to generate BGM 410, and BGM generation function 944a generates BGM 410 based on the instruction, input information 100, generation policy 200, and video 320.

[0256] In S212, upon completion of the generation of BGM 410 by the BGM generation function 944a, the command function 960 instructs the editing function 950 to generate multimodal content 800, and the editing function 950 edits the video 320 and BGM 410 based on the generation policy 200 to generate multimodal content 800.

[0257] In the above description, the production of multimodal content 800 is described in a manner in which the production is dynamically managed by the command function 960, but the present invention is not limited thereto. For example, the present invention may include a manner in which each process, such as the generation of visual information 300, the generation of audio information 400, and the editing to integrate them, is executed sequentially in a predetermined order based on the generation policy 200 (so-called batch processing). In other words, the command function 960 does not necessarily require dynamic adjustment or changes in the execution order between each process, and may simply function as a batch processing control unit that starts the next process in response to the completion of one process and integrates the outputs of each process to produce multimodal content 800. Even in this case, the command function 960 can be said to manage the execution status of multiple functions in order to determine the completion of one process and control the start of the execution of subsequent processes.

[0258] Figure 9 schematically shows an example of the hardware configuration of a computer 1200 that functions as a server on which an AI agent is implemented, which realizes various functions such as an information processing device 900, a storage device 909, and a command function 960.

[0259] In this embodiment, generative AI refers to artificial intelligence (including AGI (Artificial General Intelligence) or ASI (Artificial Superintelligence)) that generates content such as text, images, audio, and video using deep learning technologies such as transformers, self-attention, and autoregressive networks, including, for example, generative AI, generative systems, language models (LLM (Large Language Model), SLM (Small Language Model)), GPT (Registered Trademark) (Generative Pre-trained Transformer), Gemini (Registered Trademark), Claude (Registered Trademark), Llama (Registered Trademark), and other language models. Extension technologies for generative AI include frameworks such as Retrieval Augmented Generation (RAG), Memory Augmented Generation, Hybrid Search using Vector Databases, Chunking Processing, Knowledge Graph Linking, Entity Linking, AutoGen, AOG (Agentic Orchestration Graph), and LangChain. Techniques for improving the performance of generative AI include fine-tuning using RLHF (Reinforcement Learning from Human Feedback), RLAIF (Fine-tuning from Labeled AI Feedback), PEFT (Parameter-Efficient Fine-Tuning), LoRA (Low-Rank Adaptation), distillation, quantization, weight sharing, continuous learning, associative learning, and in-context learning.Furthermore, the generative AI can operate in any environment, including on-premise, cloud, and edge (on-device), and parallel learning, distributed learning, and inference using GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), NPUs (Neural Processing Units), IPUs (Intelligence Processing Units), ASICs (Application-Specific Integrated Circuits), and FPGAs (Field-Programmable Gate Arrays).

[0260] Generative AI forms an AI agent as a collaborative team of two or more AIs working together, cooperating, and coordinating with each other. It can be applied to an extremely wide range of application areas, including call center support, automated FAQ responses via chatbots, AI assistants, translation generation, summary generation, meeting minute generation, programming support (code generation, debugging), data analysis, system development, financial and legal document review, research, diagnostics, drug discovery, design optimization, supply chain management, education, games, metaverse, advertising, creative, marketing, e-commerce (recommendations), threat intelligence, image generation, video generation, music generation, robotics, smart factories, smart cities, traffic control, autonomous driving, and IoT device control. Use cases for generative AI include both internal processing types used in backend systems and chatbot types provided to users on the frontend.

[0261] Multiple AIs (AIs) can collaborate, cooperate, and coordinate in a chain execution manner, with an orchestrator AI or orchestration AI acting as a command center to direct the other AIs as needed, enabling the AI ​​agents to solve various tasks and challenges as a joint team. On the other hand, challenges arising from the collaboration and coordination of multiple AIs include risks such as hallucination, fake data, bias, discrimination, data leakage, misdelivery, plagiarism, privacy protection, and copyright infringement. Therefore, management and monitoring (logging and monitoring) technologies for multiple AIs are used to ensure security, governance, accountability, auditing, reliability, and compliance.

[0262] In the future, managing and operating multiple AI systems may require infrastructure such as high-performance GPUs, quantum computers, HPC (High-Performance Computing) clusters, distributed processing, parallel processing, load balancers, CDNs (Content Delivery Networks), multi-access edge computing (MEC), and fog computing. Furthermore, it is expected that the generating AI agents will achieve autonomy while balancing reliability and cost-effectiveness, and will evolve beyond the singularity to AGI or ASI levels. This invention encompasses these future technological trends and potential applications and is not limited to the embodiments described herein.

[0263] Figure 9 schematically shows an example of the hardware configuration of a computer 1200 that functions as an information processing device 900, or a computer that functions as a server or the like on which an AI agent realizing various functions such as a storage device 909 and a command function 960 is implemented. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "functions" of the device according to this embodiment, or to cause the computer 1200 to execute an operation or such one or more "functions" associated with the device according to this embodiment, and / or to cause the computer 1200 to execute a process or a stage of such process according to this embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to execute a specific operation associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0264] The computer 1200 according to this embodiment includes a CPU 1212, a GPU 1213, RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive and a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive and a solid-state drive, etc. The computer 1200 also includes legacy input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0265] The CPU 1212 operates according to the programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires the image data generated by the CPU 1212 and stores it in the frame buffer provided in RAM 1214 or within itself, so that the image data is displayed on the display device 1218.

[0266] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0267] The ROM 1230 stores boot programs and / or hardware-dependent programs of the computer 1200, which are executed by the computer 1200 upon activation. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0268] The program is provided on a computer-readable storage medium such as a DVD-ROM or IC card. The program is read from the computer-readable storage medium and installed on a storage device 1224, RAM 1214, or ROM 1230, which are examples of computer-readable storage media, and executed by the CPU 1212. The information processing described within these programs is read by the computer 1200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the operation or processing of information in accordance with the use of the computer 1200.

[0269] For example, when communication is performed between a computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into RAM 1214 and, based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as RAM 1214, storage device 1224, DVD-ROM, or IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area provided on the recording medium.

[0270] Furthermore, the CPU 1212 may read all or necessary parts of a file or database stored on an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), or an IC card into the RAM 1214, and perform various types of processing on the data in the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

[0271] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 1212 may perform various types of processing on the data read from RAM 1214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to RAM 1214. The CPU 1212 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 1212 may search among the multiple entries for an entry that matches the specified condition for the attribute value of the first attribute, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies the predetermined condition.

[0272] The program or software module described above may be stored on or near the computer 1200 in a computer-readable storage medium. Alternatively, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0273] In this embodiment, blocks in the flowchart and block diagram may represent stages in a process in which an operation is performed or "functions" of devices that have the role of performing an operation. Specific stages and "functions" may be implemented by dedicated circuits, programmable circuits supplied with computer-readable instructions stored on a computer-readable storage medium, and / or processors supplied with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuits may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits, such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, exclusive OR, negated AND, negated OR, and other logical operations, flip-flops, registers, and memory elements.

[0274] A computer-readable storage medium may include any tangible device capable of storing instructions that can be executed by a suitable device, and as a result, a computer-readable storage medium having instructions stored therein will comprise a product that includes instructions that can be executed to create means for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital multipurpose disc (DVD), Blu-ray® disc, memory stick, integrated circuit card, etc.

[0275] Computer-readable instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, Java®, C++, and traditional procedural programming languages ​​such as the C programming language or similar programming languages.

[0276] Computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the internet to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or another programmable data processing device, so that the processor or programmable circuit of the programmable data processing device, such as a computer, may execute the instructions to generate means for performing operations specified in a flowchart or block diagram. Here, the computer may be a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, a general-purpose computer, or a special-purpose computer, and may also be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system and is a computer in a broad sense. In a distributed computing system, multiple computers execute a program collectively by each computer executing a part of the program and passing data during program execution between computers as needed.

[0277] Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, and microcontrollers. A computer may have one or more processors. In a multiprocessor system with multiple processors, each processor executes a portion of the program, and the processors collectively execute the program by passing program execution data between them as needed. For example, in the execution of multitasking, each of the multiple processors may execute a portion of each task in small chunks by switching tasks at each time slice. In this case, which part of a program each processor executes changes dynamically. Which part of a program each of the multiple processors executes may also be statically determined by multiprocessor-aware programming.

[0278] The invention according to this embodiment provides a means to produce content that faithfully reflects the user's creative intent and physical constraints in a short period of time and at low cost. Therefore, it can at least contribute to Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0279] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0280] It should be noted that the execution order of operations, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before," "prior to," etc., and that these can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," "next," etc. for convenience, it does not mean that it is essential to perform the operations in that order. [Explanation of Symbols]

[0281] 10 User, 90 Information Processing System, 97 Communication Channel, 99 Network, 100 Input Information, 110 Intent Information, 120 Material Information, 122 Image, 124 Text Data, 130 Reference Content, 200 Generation Policy, 210 Constraints, 212 Format Policy, 214 Content Policy, 220 Direction Policy, 300 Visual Information, 310 Storyboard, 320 Video, 400 Audio Information, 410 BGM, 420 Narration, 500 Text Information, 800 Multimodal Content, 900 Information Processing Device, 909 Storage Device, 910 Information Reception Function, 920 Analysis Function, 930 Policy Generation Function, 940 Element Generation Function, 942 Visual Information Generation Function, 942a Storyboard Generation Function, 942b Video Generation Function, 944 Audio Information Generation Function, 944a BGM generation function, 944b Narration generation function, 946 Text information generation function, 950 Editing function, 960 Command function, 970 Change management function, 980 Quality check function, 990 Output control function, 1200 Computer, 1210 Host controller, 1212 CPU, 1213 GPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 Input / Output controller, 1222 Communication interface, 1224 Storage device, 1230 ROM, 1240 Input / Output chip

Claims

1. An information processing system that generates multimodal content, A policy generation function that generates a generation policy including constraints for generating the multimodal content and performance policies regarding performance guidelines for generating the multimodal content, based on input information regarding the subject of the multimodal content to be generated, An element generation function that generates at least two pieces of information from among visual information relating to the visual elements of the multimodal content, audio information relating to the audio elements, and text information relating to the text elements, based on the aforementioned generation policy. An editing function that generates the multimodal content by editing the at least two pieces of information in accordance with the constraints based on the generation policy, A command function that manages the execution status of the generation of the generation policy by the policy generation function, the generation of at least two of the visual information, audio information, and text information by the element generation function, and the generation of the multimodal content by the editing function. It has multiple functions, including Information processing system.

2. The information processing system according to claim 1, wherein the policy generation function generates the generated policy by adjusting the theatrical guidelines to the extent that the constraints are satisfied.

3. The aforementioned constraints include a formal policy that defines constraints on the formal aspects of the multimodal content, and a content policy that defines constraints on the content that the multimodal content should include. The information processing system according to claim 2, wherein the policy generation function generates the generated policy by adjusting the content policy to the extent that the formal policy is satisfied, and by adjusting the presentation policy to the extent that the formal policy and the content policy are satisfied.

4. The aforementioned constraints include a formal policy that defines formal constraints of the multimodal content and a content policy that defines content constraints that the multimodal content should include. The information processing system according to claim 2, wherein the policy generation function manages to generate the generated policy by prohibiting the generation of the content policy and the generation of the performance policy until the generation of the format policy is completed, and prohibiting the generation of the performance policy until the generation of the content policy is completed.

5. The information processing system according to claim 1, wherein the element generation function includes a format policy among the generation policy that defines the constraints on the format of the multimodal content, in the generated product which has been generated from at least two of the visual information, the audio information, and the text information.

6. The input information includes intent information representing the production intent of the multimodal content and material information relating to the materials of the multimodal content. The information processing system according to claim 1, wherein the policy generation function generates the generation policy based on the intent information and the material information.

7. The input information includes intent information representing the production intent of the multimodal content and images relating to the materials of the multimodal content. The information processing system further includes an analysis function that analyzes the image and obtains structured text data from the image. The information processing system according to claim 1, wherein the policy generation function generates the generation policy based on structured text data of the intent information and the image.

8. The information processing system according to claim 7, wherein the policy generation function manages to prohibit the generation of the generation policy until the analysis function has completed acquiring the structured text data of the image, and then generates the generation policy.

9. The aforementioned input information includes images relating to the materials of the multimodal content, The information processing system according to claim 1, wherein the policy generation function generates the generation policy which includes a policy prohibiting at least one attribute of the subject included in the image from changing in violation of natural laws between the start and end points of the multimodal content, such as shape, color, components, logo, and price.

10. The aforementioned input information further includes content that serves as a reference for generating the multimodal content, The policy generation function generates the generated policy based on the reference content, The information processing system according to claim 1, wherein the element generation function generates at least two of the visual information, audio information, and text information based on the reference content.

11. An information receiving function that receives the aforementioned input information from the user, The input information received by the information receiving function, the generated policy generated by the policy generation function, the at least two pieces of information generated by the element generation function, and the change management function that manages whether or not the multimodal content generated by the editing function can be changed. The information processing system according to claim 1, further comprising the following:

12. The information processing system according to claim 1, wherein the command function manages the start and completion of the execution of the plurality of functions.

13. The information processing system according to claim 12, wherein the command function starts the execution of the policy generation function, determines the execution order of the plurality of functions based on the generated policy generated by the policy generation function upon completion of the execution of the policy generation function, starts the execution of one of the plurality of functions according to the determined execution order, and manages to start the execution of the next function in the determined execution order upon completion of the execution of that one function.

14. The system further includes a quality check function that outputs a content evaluation result by performing at least one of the following evaluations: evaluating the product generated by the policy generation function based on the input information; evaluating the product generated by the element generation function based on at least one of the input information and the generation policy; and evaluating the product generated by the editing function based on at least one of the input information and the generation policy. The information processing system according to claim 1, wherein the command function adjusts the execution status based on the content evaluation results.

15. The multimodal content is video content including the visual information, the audio information, and the text information. The aforementioned element generation function is A storyboard generation function that generates storyboard information from the aforementioned visual information, A video generation function that generates video information from the aforementioned visual information, A BGM generation function that generates BGM (Background Music) from the aforementioned audio information, A narration generation function that generates narration from the aforementioned audio information, It has, The input information includes intent information representing the production intent of the video content and material information relating to the materials of the video content. The aforementioned command function manages the start and completion of execution of multiple functions provided by the information processing system, The policy generation function generates the generated policy based on the intent information and the material information in response to the execution start process by the command function. The information processing system according to claim 1, wherein the command function, upon completion of the execution of the policy generation function, determines the execution order of a plurality of functions, including the storyboard generation function, the video generation function, the BGM generation function, and the narration generation function, based on the generated policy generated by the policy generation function, starts the execution of one of the plurality of functions according to the determined execution order, and manages to sequentially execute the process of starting the execution of the next function in the determined execution order upon completion of the execution of the said one function.

16. The input information includes intent information that represents the intention behind the creation of the multimodal content. The information processing system according to claim 1, wherein the policy generation function converts the information expressed from the perspective of the producer of the multimodal content into information expressed from the perspective of the viewer of the multimodal content, and generates the generation policy based on the information expressed from the perspective of the viewer.

17. The information processing system according to claim 1, wherein the element generation function classifies emotional elements to be included in the multimodal content into multiple energy levels based on the generation policy, and controls at least one of the volume, tempo, and beat of the audio information, and the presence or absence of a human voice, based on the energy level.

18. The information processing system according to claim 1, wherein the element generation function generates background music included in the audio information under the constraint that it becomes a single, consistent background music from the start to the end of the multimodal content, based on the generation policy.

19. Equipped with multiple AI (Artificial Intelligence) agents, The multiple AI agents realize the multiple functions, The information processing system according to any one of claims 1 to 18, wherein the AI ​​agent that implements the command function and the AI ​​agent that implements the element generation function are different among the plurality of AI agents.

20. The information processing system according to claim 19, wherein each of the policy generation function, element generation function, editing function, and command function is implemented by an AI agent, and the AI ​​agents that implement each of the policy generation function, element generation function, editing function, and command function are different from each other.

21. A computer-based information processing method for generating multimodal content, The generation of the generation policy by a policy generation function that generates a generation policy including constraints for generating the multimodal content and performance policies for performance guidelines for generating the multimodal content, based on input information regarding the subject of the multimodal content to be generated, Based on the aforementioned generation policy, an element generation function generates at least two of the following information: visual information relating to the visual elements of the multimodal content, audio information relating to the audio elements, and text information relating to the text elements; A command stage that manages the execution status of the generation of the multimodal content by an editing function that generates the multimodal content by editing the at least two pieces of information in accordance with the constraints based on the generation policy. Equipped with Information processing methods.

22. A program for causing a computer to execute the information processing method described in claim 21.

Images