Image generation system, image generation method, and image generation program
The video generation system addresses the burden of manual image preparation by using AI to generate videos from user queries, enhancing usability and efficiency in video creation.
Patent Information
- Application Number
- JP2024065475
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-04-15
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2044-04-15
Smart Images

Figure 0007578209000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image generation system, an image generation method, and an image generation program. [Background technology]
[0002] There are techniques for automatically generating images (also called "videos"), such as a technique for estimating a three-dimensional posture of a virtual person from a two-dimensional line drawing and generating a video (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2003-058906 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, there is room for improvement in the conventional technology. For example, in the conventional technology, a two-dimensional line drawing, i.e., an image, is required to generate a video, and preparing an image such as a two-dimensional line drawing places a large burden on the user, and it is difficult to generate a video when the user cannot prepare an image. Therefore, it is desired to provide a video generation service that places a small burden on the user and has high usability, and it is desired to obtain video data in response to an input query from the user, for example.
[0005] Therefore, the present disclosure proposes an image generation system, an image generation method, and an image generation program capable of acquiring video data in response to a query input by a user. [Means for solving the problem]
[0006] In order to solve the above problems, one embodiment of a video generation system according to the present disclosure includes an acquisition unit that acquires an input query related to video generation from a user, a scenario generation unit that generates scenario data related to video generation based on the input query, a code generation unit that generates code for constructing 3D data based on the scenario data, and a video data acquisition unit that acquires video data based on the code. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 illustrates an example of an image generation system according to the present disclosure. [Diagram 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image generation system according to the present disclosure. [Diagram 3] FIG. 1 is a diagram showing an example of a flow of an image generation process according to the present disclosure. [Figure 4] FIG. 11 is a diagram showing another example of the flow of the image generation process of the present disclosure. [Diagram 5] FIG. 11 is a diagram showing an example of a flow of an evaluation process according to the present disclosure. [Figure 6] FIG. 11 is a diagram illustrating an example of a generation process of scenario generation information. [Figure 7] FIG. 11 is a diagram illustrating an example of a generation process of scenario data. [Figure 8] FIG. 11 is a diagram illustrating an example of a process for generating information for code generation. [Figure 9] FIG. 11 is a diagram illustrating an example of image quality improvement processing. [Figure 10] FIG. 4 illustrates an example of a user interface. [Figure 11] FIG. 4 illustrates an example of a user interface. [Figure 12] FIG. 4 illustrates an example of a user interface. [Figure 13] FIG. 4 illustrates an example of a user interface. [Figure 14] FIG. 4 illustrates an example of a user interface. [Figure 15] FIG. 4 illustrates an example of a user interface. [Figure 16] FIG. 4 illustrates an example of a user interface. [Figure 17] FIG. 11 is a diagram illustrating an example of a generation process of information for sound generation. [Figure 18] FIG. 1 illustrates an example of a USD file. [Figure 19] 13 is a flowchart showing a processing procedure executed by the video production system. [Figure 20] FIG. 4 illustrates an example of a user interface. [Figure 21] FIG. 4 illustrates an example of a user interface. [Figure 22] FIG. 4 illustrates an example of a user interface. [Diagram 23] FIG. 11 illustrates an example of an editing process. [Figure 24] FIG. 11 illustrates an example of an editing process. [Diagram 25] FIG. 11 illustrates an example of an editing process. [Figure 26] FIG. 11 illustrates an example of an editing process. [Figure 27] FIG. 11 illustrates an example of an editing process. [Figure 28] FIG. 11 illustrates an example of an editing process. [Figure 29] FIG. 11 illustrates an example of an editing process. [Diagram 30] FIG. 13 is a diagram showing an example of a confirmation operation during editing. [Diagram 31] 13 is a flowchart showing a processing procedure executed by the video production system. [Diagram 32] FIG. 13 is a diagram illustrating an example of an evaluation process according to a range selection. [Diagram 33] FIG. 11 is a conceptual diagram showing an example of processing according to the depth of field. [Diagram 34] FIG. 13 is a diagram showing an example of an image during a confirmation operation. [Diagram 35] FIG. 13 is a diagram showing an example of highlighting of a changed portion. [Diagram 36] FIG. 13 is a diagram showing an example of presentation of the relationship between cuts. [Figure 37] FIG. 13 is a diagram illustrating an example of object selection. [Figure 38] FIG. 13 is a diagram illustrating an example of object selection. [Figure 39] FIG. 13 is a diagram illustrating an example of processing using reference data. [Diagram 40] 10 is a flowchart showing a flow of processing in response to a user operation. [Diagram 41] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the image generation system, image generation method, and image generation program according to the present application are not limited to these embodiments. In addition, in the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.
[0009] The present disclosure will be described in the following order. 1. Embodiment 1-1. Overview of the configuration of the image generation system of the present disclosure 1-2. Processing by the image generation system of the present disclosure 1-3.User Interface 1-4. Processing example 1-4-1. Sound generation example 1-4-2. Example of text logo generation 1-4-3. Re-learning example 1-4-4.USD update example 1-4-5. Evaluation example 1-4-6. Example of multiple cut selection 1-4-7.Examples of using time information 1-4-8. Response examples according to the degree of verbalization 1-4-9. Examples of using qualitative values 1-4-10. Example of rendering process during input 1-4-11. Example of confirmation process when editing 1-4-12. Example of processing according to range selection 1-4-13. Examples of processing according to depth of field 1-4-14. Example of playback processing during confirmation work 1-4-15. Highlighting example 1-4-16. Example of showing the relationship between cuts 1-4-17. Example of object selection 1-4-18.Examples of using 3D models 1-4-19. Examples of using reference data 1-4-20. Examples of benefits of having 3D data 1-5. Example of processing flow from the user's perspective 1-6.About AI models 2. Other embodiments 2-1.Other configuration examples 2-2.Other 3. Effects of the present disclosure 4. Hardware Configuration
[0010] <1. Embodiment> <1-1. Overview of the configuration of the image generation system of the present disclosure> Fig. 1 is a diagram showing an example of an image generation system according to the present disclosure. The image generation system 1 includes an image generation module 100, an information acquisition module 200, a sensor unit 300, and a client UI display unit 400. Although Fig. 1 shows only one of each configuration, the image generation system 1 may include multiple image generation modules 100, multiple information acquisition modules 200, multiple sensor units 300, and multiple client UI display units 400.
[0011] First, a description will be given of the configuration of the image generation module 100 that performs image generation processing. The image generation module 100 includes an input text analysis unit 110, a sensor analysis unit 120, a prompt generation unit 130, an image generation unit 140, a sound generation unit 150, a text / logo generation unit 160, a composite editing unit 170, an evaluation unit 180, a client UI module 190, and the like.
[0012] The input text analysis unit 110 analyzes the input text. For example, the input text analysis unit 110 analyzes the text input from the information acquisition module 200. The sensor analysis unit 120 analyzes the input sensor information. For example, the sensor analysis unit 120 analyzes the sensor information acquired from the information acquisition module 200.
[0013] The prompt etc. generation unit 130 generates various information required for generating a video (video) including a prompt etc. to be input to an AI (Artificial Intelligence) model (also simply referred to as a "model"), which is a machine learning model described later. For example, the prompt etc. generation unit 130 generates a prompt using a user's input and a pre-stored prompt (template etc.). Note that a prompt is merely one example of information to be input to an AI model (model input information), and the model input information to be input to an AI model is not limited to a prompt, and any format of model input information can be adopted, and the "prompt etc. generation unit" may be read as a "model input information etc. generation unit." In FIG. 1, the prompt etc. generation unit 130 has a scenario-oriented generation unit 131, a video-oriented generation unit 132, a sound-oriented generation unit 133, and a text / logo-oriented generation unit 134.
[0014] The scenario-oriented generation unit 131 generates various information related to the generation of a scenario. The scenario-oriented generation unit 131 generates input information to be input to a model that outputs a scenario. For example, the scenario-oriented generation unit 131 is a scenario generation unit that generates scenario data related to video generation based on an input query. For example, the scenario-oriented generation unit 131 is a first output unit that outputs scenario generation information used by the scenario generation unit to generate scenario data based on an input query.
[0015] The video generation unit 132 generates various information related to the generation of videos. The video generation unit 132 generates input information to be input to a model that outputs code for configuring 3D (three-dimensional) data. For example, the video generation unit 132 is a code generation unit that generates code for configuring 3D data based on scenario data. For example, the video generation unit 132 is a second output unit that outputs code generation information used by the code generation unit to generate code for configuring 3D data based on scenario data.
[0016] The sound-oriented generation unit 133 generates various information related to the generation of sound information. The sound-oriented generation unit 133 generates input information to be input to a model that outputs sound. The text / logo-oriented generation unit 134 generates various information related to the generation of text and logos. The text / logo-oriented generation unit 134 generates input information to be input to a model that outputs at least one of text and logos.
[0017] The video generating unit 140 executes processing related to the generation of video. The video generating unit 140 is a video acquiring unit that acquires video data based on a code. The video generating unit 140 generates video using various information generated by the prompt generating unit 130. For example, the video generating unit 140 is a video generating unit that generates video data based on a code. The video generating unit 140 may acquire video data in any manner. For example, the video generating unit 140 may acquire video data by transmitting data used for generating video data to an external service providing device (vendor, etc.) that provides a service of generating video data, and receiving the video data generated by the service providing device from the service providing device. In FIG. 1, the video generating unit 140 has a USD generating unit 141, a rendering unit 142, and a video refining unit 143.
[0018] The USD generation unit 141 generates various information related to a universal scene description (USD). For example, the USD generation unit 141 generates USD-Python or the like by an AI model such as a Large Language Model (hereinafter also referred to as "LLM") using a prompt obtained by the video prompt generation in the video generation unit 132.
[0019] The rendering unit 142 executes various processes related to rendering. The rendering unit 142 executes a process of rendering the USD generated by the USD generation unit 141.
[0020] The image refinement unit 143 executes various processes for refining the image. The rendering unit 142 improves the quality of the generated image by image refinement processing. For example, the image refinement unit 143 is an image quality improvement unit that executes image quality improvement processing to improve the image quality of video data.
[0021] The sound generation unit 150 executes a process of generating sounds. The sound generation unit 150 generates sound information such as BGM (background music), SE (sound effect), narration, and dialogue, using a prompt obtained by the sound-oriented prompt generation in the sound-oriented generation unit 133, with an AI model such as a contrastive learning model.
[0022] The text / logo generating unit 160 executes a process for generating at least one of a text and a logo. The text / logo generating unit 160 generates at least one of a text and a logo using the information generated by the text / logo generating unit 134.
[0023] With the above-described configuration, the image generation module 100 generates a prompt for generating a scenario by combining a pre-stored prompt with a user's input. The image generation module 100 generates a scenario by inputting the generated prompt into an AI model such as an LLM. The image generation module 100 also generates a prompt for generating an image, sound, and text / logo from the generated scenario. The image generation module 100 performs image generation, sound generation, and text / logo generation using prompts, scenarios, and the like for generating an image, sound, and text / logo.
[0024] The composite editing unit 170 executes processes related to editing. For example, the composite editing unit 170 executes a process of combining (combining) the generated video, sound, and text / logo into one video.
[0025] The evaluation unit 180 executes an evaluation process for evaluating various targets. The evaluation unit 180 evaluates the information generated by the above-mentioned configuration. For example, the evaluation unit 180 generates information indicating an evaluation of at least one of the scenario data and the video data.
[0026] The client UI module 190 executes processing related to output in a UI (User Interface) on the client side. For example, the client UI module 190 generates various information related to output in a UI on the client side. In this case, the client UI module 190 executes processing to generate a UI to be displayed on the user side. The client UI module 190 generates various information to be displayed on the client UI display unit 400.
[0027] Furthermore, the information acquisition module 200 acquires various pieces of information. The information acquisition module 200 has an input text acquisition unit 210, a sensor acquisition unit 220, and the like. The input text acquisition unit 210 acquires text information inputted via the keyboard 320 or the microphone 330. For example, the input text acquisition unit 210 acquires text information inputted by a user via the keyboard 320 or the microphone 330. For example, the input text acquisition unit 210 is an acquisition unit that acquires an input query related to video generation from a user.
[0028] The sensor acquisition unit 220 acquires information (also referred to as "sensor information") detected by a sensor such as a camera 340 or a motion capture device. The information acquisition module 200 provides (transmits) the acquired various information to the image generation module 100. Note that the information acquisition module 200 may be integrated with the image generation module 100.
[0029] The sensor unit 300 has various sensors. The sensor unit 300 senses user input. The sensor unit 300 accepts operations by the user. For example, the sensor unit 300 is a reception unit that accepts operations related to video editing from the user. For example, the sensor unit 300 has a mouse 310, a keyboard 320, a microphone 330, a camera 340, an IMU 350 which is an inertial measurement unit, and the like. In this way, the sensor unit 300 includes a user terminal (such as a smartphone) equipped with the microphone 330, the camera 340, and the IMU 350, as well as sensors such as a motion capture device, in addition to the mouse 310 and the keyboard 320, and senses user input.
[0030] The client UI display unit 400 displays various information to be presented to a client (user). The client UI display unit 400 displays the UI generated by the client UI module 190 on a display (display device) of the client. For example, the client UI display unit 400 is a display control unit that displays a storyboard based on scenario data. The storyboard is configured to display video data for each cut of the video.
[0031] The video generation system 1 may have a hardware configuration as shown in Fig. 2. Fig. 2 is a diagram showing an example of a hardware configuration of the video generation system of the present disclosure. In Fig. 2, the video generation system 1 has, as a hardware configuration, a cloud-side computer 10, a client-side computer 20, a camera / sensor 30 including various sensors such as a camera, and the like. The video generation system 1 may also include an information providing device (computer) that provides information resources 40 such as learning data and an AI model 50 to the computer 10.
[0032] 2 is merely an example, and any hardware configuration can be adopted for the image generation system 1 as long as the desired processing can be executed. For example, the computer 10 and the computer 20 may be integrated. Furthermore, the information resource 40 and the AI model 50 may be stored inside the computer 10.
[0033] The computer 10 includes a CPU (Central Processing Unit) 11, a GPU (Graphics Processing Unit) 12, a communication device 13, and a memory / storage 14. For example, the computer 10 corresponds to the image generation module 100 and the information acquisition module 200 in Fig. 1. The computer 10 may be a service providing device (server device) that provides an image generation service. The CPU 11 and the GPU 12 are so-called processors, and execute calculation processing (arithmetic processing) related to various processes such as image generation.
[0034] The communication device 13 is a communication device having a communication function for transmitting and receiving information between the computer 20, an information providing device, etc., and may be, for example, a communication circuit, a NIC (Network Interface Card), etc. The communication device 13 communicates with other devices such as the computer 20 and the information providing device via a predetermined network (such as the Internet). For example, the communication device 13 is connected to the predetermined network by wire or wirelessly, and transmits and receives information between the computer 20, the information providing device, etc.
[0035] The memory / storage 14 is a storage device that stores various types of information. The memory / storage 14 is, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The memory / storage 14 stores various types of information used for processing by processors such as the CPU 11 and the GPU 12. The memory / storage 14 may store information resources 40, an AI model 50, etc.
[0036] The computer 20 includes a CPU 21, a GPU 22, a communication device 23, a memory / storage 24, and an IO interface 25. For example, the computer 20 corresponds to the client UI display unit 400 in FIG. 1. The computer 20 may be a terminal device (such as a personal computer (PC) or a mobile device such as a smartphone) used by a user who uses the video generation service. The CPU 21 and the GPU 22 are so-called processors, and execute calculation processing (arithmetic processing) related to various processes such as video display. Note that the above is merely an example, and the computer 20 may have any configuration as long as it can perform the desired processing. For example, the computer 20 may execute calculation processing (arithmetic processing) related to various processes such as video display using a circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The computer 20 may be configured to directly incorporate a program into the circuit of the processor instead of storing the program in a memory (such as the memory / storage 24). In this case, the processor realizes the function by reading and executing the program incorporated in the circuit. In addition, each processor in the present embodiment is not limited to being configured as a single circuit, but may be configured as a single processor by combining multiple independent circuits to realize its functions. Also, like the computer 20, the computer 10 may have any configuration as long as it can perform the desired processing.
[0037] The communication device 23 is a communication device having a communication function for transmitting and receiving information to and from the computer 10, the sensor 30, etc., and may be, for example, a communication circuit, a NIC, etc. The communication device 23 communicates with other devices such as the computer 10, the sensor 30, etc. via a predetermined network (such as the Internet). For example, the communication device 23 is connected to the predetermined network by wire or wirelessly, and transmits and receives information to and from other devices such as the computer 10, the sensor 30, etc.
[0038] The memory / storage 24 is a storage device that stores various types of information. The memory / storage 24 is, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The memory / storage 24 stores various types of information used for processing by processors such as the CPU 21 and the GPU 22.
[0039] The IO interface 25 is an input / output interface device. The computer 20 receives input from the sensor 30 via the IO interface 25. For example, the computer 20 receives input from an input device such as a keyboard or a mouse via the IO interface 25. The computer 20 also outputs information from a display (display device) and a speaker (audio output device) via the IO interface 25. For example, the computer 20 plays video on the display and the speaker via the IO interface 25.
[0040] Various sensors 30 such as cameras sense user input. Various sensors 30 such as cameras accept operations by the user. For example, the sensor 30 corresponds to the sensor unit 300 in FIG. 1. The information resource 40 includes various information such as learning data. For example, the information resource 40 includes learning data used for learning various AI models such as LLM. The AI model 50 includes information on an AI model used in processing related to image generation such as LLM. For example, the AI model 50 includes information on various AI models such as models M1 to M3 described later. As described above, the image generation system 1 may have a configuration other than that shown in FIG. 2.
[0041] <1-2. Processing by the image generation system of the present disclosure> From here, the processing by the video generation system will be described. First, an example of the flow of the video generation processing shown in Fig. 3 will be described. Fig. 3 is a diagram showing an example of the flow of the video generation processing of the present disclosure. Note that the processing described below with the video generation system 1 as the processing subject may be performed by any device capable of executing the processing, depending on the device configuration included in the video generation system 1.
[0042] User input information UIN1, denoted as "User's Input" in FIG. 3, corresponds to information input by a user for image generation (also referred to as "input query"). Note that the input query is not limited to text (character information) and any information can be adopted. The input query may be any information including at least one of text, image, audio, and 3D data.
[0043] The video generation system 1 uses the user input information UIN1 to generate scenario generation information (also referred to as "first input information") to be used as an input for model M1, denoted as "LLM" in FIG. 3, which will be described later. For example, model M1 is a first model that outputs scenario data in response to the input of the first input information. Any AI model such as an LLM (large scale language model) can be used for model M1 as long as it is capable of producing the desired output in response to the input. AI models such as model M1 will be described later.
[0044] The video generation system 1 generates a scenario FD1 by inputting first input information to the model M1 and causing the model M1 to output a scenario FD1, which is scenario data. The video generation system 1 then generates USD generation necessary information SD1, which is code generation information (also referred to as "second input information") used as an input for the model M3, using the scenario FD1, the output of the model M2 that uses the scenario FD1 as an input, and user input information UIN2, etc. For example, the model M3 is a second model that outputs a code in response to the input of the second input information. The model M3 can employ any AI model, such as an LLM (large-scale language model), as long as it is capable of producing a desired output in response to the input.
[0045] 3 shows only one piece of information SD1 required for USD generation, but there may be multiple pieces of information SD1 required for USD generation depending on, for example, the number of USD files to be generated. For example, there may be multiple pieces of information SD1 required for USD generation depending on the number of USD files to be generated corresponding to the data structure shown in FIG.
[0046] For example, the model M2 may be a model that outputs a template or the like corresponding to the scenario data in response to the input of the scenario data. Any AI model may be adopted for the model M2 as long as it is capable of providing a desired output in response to the input. For example, the user input information UIN2 may be information for specifying constraint conditions in image generation. Note that the image generation system 1 may generate second input information using the scenario FD1 and template input information, but this point will be described later.
[0047] The video generation system 1 inputs information SD1 required for USD generation to the model M3, and generates the Python code OD1 by having the model M3 output the Python code OD1, indicated as "python" in FIG. 3. For example, the Python code OD1 is (program) code that generates data in USD format (also called a "USD file") when executed. Note that Python is only an example, and any code format, not limited to Python, can be used as long as it can generate desired data for 3DCG. Also, USD is only an example, and any format, such as FBX (Film Box), can be used as long as the data is for 3DCG. The video generation system 1 executes the Python code OD1 to generate a USD file OD2, indicated as "USD" in FIG. 3.
[0048] The video generation system 1 executes a rendering process PS1, which is denoted as "Renderer" in Fig. 3, to generate video data MV1, which is denoted as "PreMovie" in Fig. 3. For example, the video data MV1 is data (also referred to as "first video data") before executing a refinement process PS2, which will be described later.
[0049] The video generation system 1 executes a refinement process PS2, which is indicated as "Refiner" in Fig. 3, to generate video data MV2, which is indicated as "RefinedMovie" in Fig. 3. For example, the refinement process PS2 is a picture quality improvement process that improves the picture quality of the video data. The video data MV2 is data (also referred to as "second video data") obtained after the video data MV1, which is the first video data, is updated by the refinement process PS2.
[0050] The video production system 1 executes composite editing PS3 using the video data MV2, user input information UIN3, etc., to generate video data MV3 indicated as "FinalMovie" in Fig. 3. For example, the composite editing PS3 executes a process of updating (editing) the video data MV2 in response to a user editing instruction indicated by the user input information UIN3, thereby generating video data MV3 in which the video data MV2 has been updated.
[0051] The flow of the video generation process shown in FIG. 3 is merely an example, and the video generation system 1 can adopt any processing mode as long as it can generate video data from a user's input query. For example, in FIG. 3, the model M1 outputs a code (Python code) as an example, but the model M1 may be a model that outputs data for 3DCG such as a USD file. Furthermore, the video generation system 1 may perform various modes of video generation processing, not limited to the processing shown in FIG. 3. An example of this point will be described with reference to FIG. 4. FIG. 4 is a diagram showing another example of the flow of the video generation process of the present disclosure. FIG. 4 differs from FIG. 3 in that sound generation necessary information SD2 and text logo necessary information SD3 are generated and video generation processing is performed using them. Note that the same points as those described in FIG. 3 will not be described as appropriate.
[0052] In FIG. 4, the video generation system 1 uses a scenario FD1, the output of a model M2 that uses the scenario FD1 as an input, and user input information UIN2 to generate information required for sound generation SD2, which is information for sound generation used as an input for a model M4, denoted as "AI" in FIG. 3. For example, the model M4 is a model that outputs various sound data in response to the input of information required for sound generation SD2 and video data MV2. Any AI model can be used for the model M4 as long as it is capable of producing the desired output in response to the input. Note that the model M4 may be a model that uses only information required for sound generation SD2 as an input.
[0053] The video production system 1 inputs information SD2 required for sound production to the model M4, and generates sound data corresponding to the video by having the model M4 output sound data AD1 for BGM (Background Music), sound data AD2 for SEs, sound data AD3 for narration, etc. The video production system 1 may also generate the sound data AD1, AD2, AD3 using user input information UIN4. For example, when the model M4 outputs the sound data AD1, AD2, AD3 as one sound data, the video production system 1 may extract the sound data AD1, AD2, AD3 from the one sound data output by the model M4 based on the designation in the user input information UIN4, and generate the sound data AD1, AD2, AD3.
[0054] 4, video generation system 1 uses scenario FD1, the output of model M2 which uses scenario FD1 as input, user input information UIN2, etc. to generate information required for text logo generation SD3, which is information for generating a text logo to be used as input for model M5. For example, model M5 is a model that outputs at least one of text and a logo in response to the input of information required for text logo generation SD3. Any AI model can be used for model M5 as long as it is capable of producing the desired output in response to the input.
[0055] Video production system 1 generates text logo data corresponding to the video by inputting information SD3 required for text logo generation to model M5 and having model M5 output text logo data DI1 for Text, text logo data DI2 for Logo, etc.
[0056] Video production system 1 generates video data MV3 by executing composite editing PS3 using video data MV2, sound data AD1, AD2, AD3, text logo data DI1, DI2, user input information UIN3, etc. For example, composite editing PS3 executes a process of combining (combining) video data MV2, sound data AD1, AD2, AD3, text logo data DI1, DI2, etc. into one and generating video data MV3 as a single image.
[0057] 3 and 4 described above show an example of processing in an initial state where there is no scenario, information required for generating USD, USD, PreMovie, RefinedMovie, etc. As described above, the video generation system 1 generates a prompt for generating a scenario based on a user's input query, provides the prompt to a natural language model to generate a scenario, generates a prompt for outputting a code constituting a video from text information described in the scenario, provides the prompt to the natural language model to generate a code constituting the video, and generates a video. In this way, when creating a video, the video generation system 1 inputs what the user wants to create or the purpose, and generates an effective video storyboard and video even without knowledge of 3DCG or video production. In addition, in the video generation system 1, creating a storyboard makes it easier to edit afterwards. Details of these points will be described later.
[0058] Furthermore, the image generation system 1 may perform various processes related to image generation. For example, the image generation system 1 may perform evaluation processing on the generated information. In this regard, an example of the flow of the evaluation processing will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the flow of the evaluation processing of the present disclosure.
[0059] In FIG. 5, video generation system 1 inputs at least one of scenario FD1, information required for USD generation SD1, information required for sound generation SD2, and information required for text logo generation SD3, and causes model M10 to output evaluation text information EV1 indicating an evaluation of the input information, thereby evaluating the generated information. For example, model M10 is a model that outputs an evaluation of input information in response to the input of information. For example, model M10 outputs evaluation text indicating an evaluation of input scenario FD1 in response to the input of scenario FD1. Note that model M10 may be a model that accepts input of scenario FD1, information required for USD generation SD1, information required for sound generation SD2, and information required for text logo generation SD3, respectively, or may be a model that accepts input of a combination of these pieces of information. Model M10 may also be a model that, in response to the input of information indicating a video (such as a caption), outputs an evaluation of the video corresponding to the input information.
[0060] From here, a description will be given of the flow of the above-mentioned process, with specific examples of each process executed by the image production system 1. Note that explanations of points similar to those described above will be omitted as appropriate.
[0061] For example, the video generation system 1 generates scenario generation information (first input information) as shown in Fig. 6. Fig. 6 is a diagram showing an example of a generation process of scenario generation information. In Fig. 6, the video generation system 1 acquires user input information IDT1 and IDT2 input by the user to content CT1 as user input information. Content CT1 is content for accepting user input information for each of the questions "What kind of video do you want to create?" and "Style."
[0062] For example, the client UI display unit 400 displays the content CT1, and the sensor unit 300 accepts the user input information IDT1 and IDT2 as user input information. For example, the user input information IDT1 and IDT2 correspond to the user input information UIN1 in FIGS.
[0063] In FIG. 6, the client UI display unit 400 displays a question, "What kind of video do you want to make?". In response to the question, "What kind of video do you want to make?", the sensor unit 300 accepts user input information IDT1, "a 15-second sneaker commercial video." In addition, the client UI display unit 400 displays a question, "style." In response to the question, "style," the sensor unit 300 accepts user input information IDT2, "cinematic."
[0064] The video production system 1 may accept the user's input information in any manner, or may accept the user's selection from a plurality of candidates. For example, the video production system 1 may convert information input by the user from a keyboard or microphone into text information and accept it as the user's input information. The video production system 1 may also accept, in addition to free text, other files such as the number of seconds in the entire video, style, camerawork, and other settings, images, and videos, as the user's input information.
[0065] The video production system 1 generates a prompt PT1, which is scenario generation information (first input information), using the user input information IDT1, IDT2, and a template TP1, which is template input information. For example, the template TP1 may be preset, or may be selected from a plurality of template candidates. For example, the video production system 1 may select a template corresponding to the user's input information from among the plurality of template candidates. For example, the video production system 1 may select a template TP1 related to a movie-style advertisement from among the plurality of template candidates based on the contents indicated by the user input information IDT1, IDT2.
[0066] For example, the video production system 1 generates a prompt PT1 by reflecting the user input information IDT1 and IDT2 in the template TP1. In FIG. 6, the video production system 1 generates the prompt PT1 by adding "cinematic" indicated by the input information IDT2 to the style item of the constraint condition, and adding "15-second sneaker commercial video" indicated by the input information IDT1 to the input sentence. In this way, the video production system 1 generates a prompt for generating a scenario based on the user's input information. Note that the user's input information may be input on a single screen, or may be input by answering several questions, and examples of these points will be described later.
[0067] Moreover, the video production system 1 generates scenario data as shown in Fig. 7. Fig. 7 is a diagram showing an example of a generation process of scenario data. In Fig. 7, the video production system 1 generates scenario data SN1 using a prompt PT1. For example, the scenario data SN1 corresponds to the scenario FD1 in Figs. 3 and 4. The scenario data SN1 includes information such as the number of seconds for each scene, such as the opening scene, the scene where sneakers are put on, and an explanation of the cut.
[0068] For example, the video generation system 1 generates scenario data SN1 by inputting a prompt PT1 to a model M1 such as an LLM and outputting scenario data SN1 from the model M1. In this way, the video generation system 1 generates a scenario by inputting the generated prompt to an AI (such as an LLM). In addition to the information shown in FIG. 7 (also called "scenario information"), the scenario data SN1 also includes information such as the environment, characters, motion, camera work, lighting, and color. For example, by inputting the user's past experience learning data or learning data of a specific director or person into the model M1 or the like using RAG (Retrieval-Augmented Generation) or fine tuning so that the scenario generation has characteristics, the video generation system 1 can generate various scenario variations.
[0069] Moreover, the video production system 1 generates code generation information (second input information) as shown in FIG. 8. FIG. 8 is a diagram showing an example of a generation process of code generation information. The video production system 1 generates a prompt PT2, which is code generation information (second input information), using scenario data SN1 and a template TP2, which is template input information. For example, the template TP2 may be preset or may be selected from a plurality of template candidates. For example, the video production system 1 may select a template corresponding to a scenario from among the plurality of template candidates. For example, the video production system 1 may select a template TP2 related to a commercial from among the plurality of template candidates based on the contents indicated by the scenario data SN1.
[0070] For example, the image production system 1 generates a prompt PT2 by reflecting the scenario data SN1 in the template TP2. In Fig. 8, the image production system 1 generates the prompt PT2 by adding information indicated by the scenario data SN1 to the input sentence. In this way, the image production system 1 generates a prompt for an image based on a scenario generated by AI.
[0071] For example, FIG. 8 shows an example of prompt generation for converting a scenario related to a person into USD-Python. Conversion into USD-Python is merely one example of a conversion format, and the conversion may be any conversion format other than USD-Python. For example, the conversion format may be Python for Blender, USD, or other formats. Furthermore, the video generation system 1 may generate prompts individually for targets such as people, environments, and camerawork, or may generate prompts collectively. Furthermore, the generated prompt may include paths to assets and motions to be used, or may include source code or APIs (Application Programming Interfaces) for throwing (inputting) the assets / motions into an AI generation algorithm.
[0072] Then, the video generation system 1 generates a USD-Python file by inputting the generated prompt to an AI (such as LLM). Note that the file format is not limited to the python format, and the file may be generated in another format such as USD. The video generation system 1 then converts the file into a format that can be rendered, such as a USD file, and executes rendering to generate a PreMovie (a video file such as mp4).
[0073] The video production system 1 may perform composite editing using PreMovie, but may also perform refinement processing, which is an example of image quality improvement processing, on the PreMovie as shown in Fig. 9. Fig. 9 is a diagram showing an example of image quality improvement processing. In Fig. 9, the video production system 1 generates a second video OT1 from the first video IN1 by refinement processing using a model M11, which is a diffusion model that receives a first video IN1, which is a PreMovie, as input and outputs a second video OT1, which is a RefinedMovie.
[0074] The AI model (such as model M11) used in the refiner process is not limited to the diffusion model, and any AI model such as LDM (Latent Diffusion Model) or LCM (Latent Consistency Model) can be used. Furthermore, the refiner process may use techniques such as AnimateDiff (time direction stabilization) and ControlNet (line art control). By using such refiner process, the video generation system 1 can improve the quality of the video while maintaining the consistency of the characters, backgrounds, props, etc.
[0075] Furthermore, the refiner process may use prompts in addition to videos. For example, the model M11 may input prompt IN2 in addition to the first video IN1. For example, when a target such as a woman in her 30s is specified by prompt IN2, the model M11 outputs a second video OT1 in which the part of the first video IN1 that represents a woman in her 30s has been improved. This allows the video generation system 1 to generate a second video OT1 in which the image quality, etc., of the target specified by prompt IN2 in the first video IN1 has been improved.
[0076] From the user's perspective, the above-mentioned processing of the video production system 1 appears to generate a scenario (storyboard) and video of each cut after input, and the processing in between is closed within the system. These processes may be generated all at once from the user's input text, from the scenario to all the cuts and refinement processing, or the user may input a preference during the processing. For example, the video production system 1 may generate several scenarios with only the outline, select one from the scenario, and execute detailed scenario and video generation processing based on the selected outline. Also, the video production system 1 may generate several patterns of characters to be generated in the video before video generation after the scenario is generated, and execute rendering processing and refinement processing of the video after the user selects one.
[0077] The components of a scenario may include videos of cuts, representative images (such as the first frame of a video), explanations of the cuts, characters (visuals, settings, etc.), the motions of each character, lighting, camera work, background environmental information, transitions between cuts, dialogue, narration, etc. Some of these are presented to the user, while others are not presented to the user and are kept for processing purposes. The scenario is arranged in chronological order by cut.
[0078] Currently, various existing video generation services are provided, such as Pika, Runway Gen-2, Lumiere, and Stable Video Diffusion. These generate videos using a diffusion model that moves vectors in the spatial and temporal directions from an image. These generate videos using only 2D images. On the other hand, video generation system 1 holds 3D information internally. For example, with existing video generation services, it is possible to modify only a specified (X,Y) area in a video, but there is an issue that the motion also changes when you want to change only the color of the clothes. On the other hand, video generation system 1 holds 3D information internally, so it is possible to modify only the targeted parts, such as only the motion, only the lighting, or only the color of a person's clothes.
[0079] In addition, existing video generation services only generate videos for each cut, and users must ensure the consistency of each cut themselves. However, video generation system 1 can consistently carry out everything from the scenario (storyboard) to video generation and editing, making it possible to generate videos with consistency in terms of actors, backgrounds, color grading, etc.
[0080] <1-3. User Interface> From here, we will describe the user interface (UI) for users who use the image generation system 1. Note that explanations of points similar to those described above will be omitted where appropriate.
[0081] As shown in FIG. 10, the video generation system 1 provides the content CT11 to the user. FIG. 10 is a diagram showing an example of a user interface. The content CT11 is a display screen (content) for receiving user input information. For example, the client UI display unit 400 displays the content CT11. The user inputs text (the "Prompt" column in FIG. 10) instructing what kind of video to create and style selection (the "Style" column in FIG. 10) as user input information via the content CT11. In this way, the user inputs the text and style selection as user input information. For example, the user inputs character information in the "Prompt" column with reference to example sentences included in the content CT11. For example, the user selects a style to be used from a plurality of style candidates displayed by pressing (clicking, etc.) the downward triangle in the "Style" column.
[0082] After completing input of the user's input information, the user selects a button labeled "Ask AI Director" in Fig. 10 to instruct the image production system 1 to generate a moving image according to the user's input information. This causes the image production system 1 to execute a process of generating a moving image according to the user's input information.
[0083] As shown in FIG. 14, the video generation system 1 provides the user with content CT15 related to the generated video. FIG. 14 is a diagram showing an example of a user interface. The content CT15 is a storyboard screen (content) for receiving user operations (instructions) for the generated video. As shown in FIG. 14, the content CT15 is a storyboard screen that displays video data for each cut of the generated video. For example, the client UI display unit 400 displays the content CT15. In this way, the video generation system 1 provides a UI that outputs a storyboard and video according to information input by the user. The user sets the video, content, narration, lines, camera work, BGM, lighting, color, and the like for each cut on the storyboard screen.
[0084] The image generation system 1 may accept user input information while asking the user a question. For example, when a button labeled "Ask AI Director" in FIG. 10 is selected, the image generation system 1 accepts user input information through a conversation (dialogue) with the user, as shown in FIGS. 11 to 13. FIGS. 11 to 13 are diagrams showing an example of a user interface. The content CT12 in FIG. 11 is a display screen (content) that presents samples generated in response to the user's input information entered in FIG. 10 and asks the user whether there is any that is close to the image. For example, the client UI display unit 400 displays the content CT12.
[0085] The content CT13 in Fig. 12 is a display screen (content) that requests (asks) detailed targets, etc. in response to the user's answer (input information) that the samples presented in the content CT12 in Fig. 11 do not match the image. For example, the client UI display unit 400 displays the content CT13.
[0086] Content CT14 in Fig. 13 is a display screen (content) that presents samples regenerated in response to a user's answer (input information) that specifically specifies a target, etc., and asks the user whether there is one that is close to the image. For example, the client UI display unit 400 displays the content CT14. Fig. 13 shows a case where the user places the mouse cursor on the leftmost sample video among the four samples and performs a specifying operation such as clicking, thereby specifying that the leftmost sample video among the four samples is the video that is close to the image. As a result, the video generation system 1 executes a process of generating a video according to the user's input information that specifies the leftmost sample video among the four samples.
[0087] In this case, the video generation system 1 provides the user with content CT15 related to the generated video, as shown in Fig. 14. For example, the client UI display unit 400 displays the content CT15. In this way, when generating a storyboard and video with the initial input contents, if necessary information is insufficient, the video generation system 1 may collect necessary information through conversation (dialogue) with the user and work out the details.
[0088] 15 and 16, the video production system 1 may have the user input a short sentence about the video to be created, generate several video stories based on the short sentence, and allow the user to select the one he or she likes. Figures 15 and 16 are diagrams showing an example of a user interface.
[0089] The video generation system 1 provides the user with the content CT21 as shown in FIG. 15. The content CT21 is a display screen (content) for receiving information input by the user. For example, the client UI display unit 400 displays the content CT21. The user inputs a sentence (short sentence) indicating what kind of video the user wants to create as the user's input information via the content CT21. For example, the user inputs character information into an input field in the content CT21.
[0090] When the user has completed inputting the user's input information, the user selects the button marked "Start" on the right edge of the input field in the content CT21 to instruct the video production system 1 to generate a video story according to the user's input information. This causes the video production system 1 to execute a process of generating a video story according to the user's input information.
[0091] As shown in Fig. 16, the video generation system 1 provides the user with content CT22 relating to the story of the generated video. The content CT22 in Fig. 16 is a display screen (content) that presents a sample of the story of the video generated corresponding to the user's input information entered in Fig. 15. For example, the client UI display unit 400 displays the content CT22. For example, if the user likes one of the video story samples, the user selects a button labeled "Continue" on the right edge of the display area for that sample, thereby instructing the video generation system 1 to generate a video corresponding to the selected sample.
[0092] If the user does not like any of the video story samples, the video production system 1 generates another pattern again. For example, if the user does not like any of the video story samples, the user selects a button labeled "Continue" on the right edge of the short sentence display area to instruct the video production system 1 to generate another pattern of video story sample again. This causes the video production system 1 to generate a video story sample of another pattern.
[0093] <1-4. Processing example> From here, in addition to the specific examples described above, specific examples of each process executed by the video production system 1 will be described. Note that explanations of points similar to those described above will be omitted as appropriate. Below, specific examples of the generation process of sounds, etc., and the evaluation process in the processing of the video production system 1 described above will be described. Note that explanations of points similar to those described above will be omitted as appropriate.
[0094] <1-4-1. Sound generation example> For example, the video production system 1 generates information for sound generation (corresponding to the information required for sound generation SD2 in FIGS. 3 and 4) as shown in FIG. 17. FIG. 17 is a diagram showing an example of a generation process of information for sound generation. The video production system 1 generates a prompt PT3, which is information for sound generation, using scenario data SN1 and a template TP3, which is template input information. For example, the template TP3 may be preset or may be selected from a plurality of template candidates. For example, the video production system 1 may select a template corresponding to a scenario from among the plurality of template candidates. For example, the video production system 1 may select a template TP3 related to a commercial from among the plurality of template candidates based on the contents indicated by the scenario data SN1.
[0095] For example, the video production system 1 generates a prompt PT3 by reflecting the scenario data SN1 in a template TP3. In Fig. 17, the video production system 1 generates a prompt PT3 by adding information indicated by the scenario data SN1 to an input sentence. For example, the video production system 1 generates a prompt for extracting several key phrases from the scenario in order to generate a sound (such as background music) that matches the scenario. The video production system 1 may input the generated prompt to an AI (such as an LLM) to obtain keywords and text information required for sound generation.
[0096] The video production system 1 generates sound data using the generated prompt PT3. For example, the video production system 1 generates sounds (sound data) such as background music, sound effects, narration, and lines based on the generated video and text information written on a storyboard (scenario data SN1, etc.). Furthermore, when necessary words (text information) such as lines and narration have already been extracted from the scenario, the video production system 1 may not generate a prompt and may instead save the words (text information) as sound data for lines, narration, etc.
[0097] For example, the video production system 1 generates sound data from the obtained information required for sound generation (information for sound generation), video, and audio data input by the user (sound source, user's voice, humming, etc.). For background music and sound effects, the video production system 1 may generate sound from text or video using a Transformer (model) such as text to music generation. The video production system 1 may also search for a contrastively learned sound source such as text to music estimation from natural text.
[0098] Furthermore, for lines and narration, the video generation system 1 may generate audio using text to speech (such as a diffusion model or flow matching) based on the words themselves and text information on the character images obtained from the scenario. Furthermore, when connecting the generated video and audio, the video generation system 1 may incorporate meta information in the video or audio file to indicate the start and end times, volume, etc. of the audio.
[0099] <1-4-2. Example of text logo generation> For example, video production system 1 generates information for generating a text logo (corresponding to information required for text logo generation SD3 in Figs. 3 and 4). Based on the generated storyboard (scenario data SN1, etc.), video production system 1 generates text and logo information (captions, titles, logos, descriptions, etc.) to be displayed on the video.
[0100] For example, the text sentence to be displayed may be clearly stated in the generated scenario, but if it is not clearly stated, the image generation system 1 generates a prompt for generating text information (text logo data, etc.) and sends it to an AI (such as an LLM) to generate the text information. Also, if the user inputs the text to be displayed, the image generation system 1 may use the information input by the user as the text information (text logo data, etc.).
[0101] The font, size, and position of the text display are determined by any method. For example, the image generation system 1 may determine the font, size, and position of the text display using any AI such as a diffusion model, a variational auto-encoder (VAE), a generative adversarial network (GAN), DALL E, StyleGAN, StyleGAN2, Pix2Pix, TransGAN, or LLM. The font, size, and position of the text display may be manually set by the user.
[0102] Regarding logos and images, the user may input images or videos in jpeg format, mp4 format, etc. In addition, regarding logos and images, the video generation system 1 may generate a prompt for image generation and throw it to any AI such as a Diffusion model, VAE, GAN, DALL E, StyleGAN, StyleGAN2, Pix2Pix, TransGAN, LLM, etc. to generate logo information.
[0103] In addition, when linking text or logo information with a video, the video production system 1 may incorporate meta information into the video or the text or logo itself to clearly indicate (meta information) the start and end times, position, and size of the text or logo.
[0104] <1-4-3. Re-learning example> Furthermore, when generating a scenario or video, the video generation system 1 may generate a scenario or video that can only be produced based on specific replaceable learning data. For example, it is possible to re-learn as learning data based on videos and images produced in the past. By re-learning with RAG or fine tuning, it is possible to change the scenario or video to be generated. In other words, the video generation system 1 can re-learn the data (history) of an individual user who has produced a video in the past, and can also generate a video using a model re-learned as learning data using the work of a specific movie director.
[0105] The learning data can be re-learned on an individual's PC or on a server. The learning data is used to train many models, such as the LLM and the Diffusion model. In cases such as when the entire scenario is to be generated using Director A, but the color of the video is to be generated using another Director B to generate color grading, the video generation system 1 may re-learn specific parts using different learning data.
[0106] <1-4-4.USD update example> As described above, the video generation system 1 modifies the 3DCG assets and rendering method that are the source of the existing video, based on the user's input information and the text information of the generated storyboard. The video generation system 1 then outputs a prompt for outputting code that composes a new video, provides the prompt to a natural language model, outputs the code that composes the video, and generates the video. This allows the video generation system 1 to modify the video in accordance with the user's input, even if the user has no knowledge of 3DCG or video production.
[0107] In the video generation system 1, after generating a video (video), scenario information, video information, etc. are saved as text or video. Therefore, the user can correct this information with input information (text, sensor, etc.).
[0108] For example, in the video production system 1, USD files are stored separately for each asset and motion, as shown in Fig. 18. Fig. 18 is a diagram showing an example of a USD file. For example, in the data structure shown in Fig. 18, a USD in a higher layer may include a path (file path) to a USD in a lower layer.
[0109] For example, the entire USD includes paths to the environment asset USD, human asset UDS, camera USD, etc. The environment asset USD also includes paths to the building asset USD and prop asset USD. The building asset USD also includes mesh information of the building itself, etc. The human asset USD also includes paths to the human mesh information and motion USD. In this way, data for 3DCG (3D data) such as a USD file may include multiple data sets. Note that the configuration (data structure) of the USD file shown in FIG. 18 is merely an example, and any configuration may be adopted, and the entire USD may be configured as one block (one data set) of USD (USD file).
[0110] When a correction is made, the image generation system 1 analyzes the user's correction information using AI (such as LLM) according to a processing flow as shown in Fig. 19, and the processing differs depending on whether the USD is replaced or part of the USD is corrected. After the correction, rendering processing and refinement processing are executed. Fig. 19 is a flowchart showing the processing procedure executed by the image generation system. As a specific example, Fig. 19 is a flowchart showing the processing procedure related to rewriting a USD file.
[0111] First, the image generation system 1 accepts correction information input by the user (step S101). For example, the sensor unit 300 accepts input information instructing the user to make corrections. The image generation system 1 analyzes the input using AI (step S102). For example, the image generation module 100 analyzes the content of the input information instructing the user to make corrections using various models, etc.
[0112] The image generation system 1 recognizes the format of the existing USD file (step S103). For example, the image generation module 100 recognizes the format of the USD file before modification. The image generation system 1 determines whether or not to modify a part of the USD (step S104). For example, the image generation module 100 determines whether or not to modify a part of the USD based on the content of input information instructing a user to modify and the format of the existing USD file.
[0113] When part of the USD is to be modified (step S104: Yes), the image generation system 1 generates a prompt for generating a modified USD-Python (step S105). For example, when part of the USD is to be modified, the image generation module 100 generates a prompt for generating a modified USD-Python by using a template or the like for generating a modified USD-Python.
[0114] The image generation system 1 generates USD-Python using AI (such as LLM) (step S106). For example, the image generation module 100 generates USD-Python by inputting a prompt into a model for generating USD-Python.
[0115] The video creation system 1 replaces the USD file to be modified (step S107). For example, the video creation module 100 replaces the USD file to be modified by reflecting the generated USD-Python in the USD file to be modified. In this manner, the video creation system 1 updates at least one of the multiple data sets of the USD file. For example, the video creation system 1 executes a process of updating some of the multiple data sets of the USD file.
[0116] The image generation system 1 executes the rendering process using the updated USD file (step S108). For example, the image generation module 100 executes the rendering process using the rewritten, i.e., corrected, USD file.
[0117] On the other hand, when the image generation system 1 does not modify any part of the USD (step S104: No), it generates a prompt for generating a creation USD-Python (step S109). For example, when the image generation module 100 does not modify any part of the USD, that is, when creating (generating) a new USD, it generates a prompt for generating a creation USD-Python using a template or the like for generating a creation USD-Python.
[0118] The image generation system 1 generates USD-Python and USD using AI (such as LLM) (step S110). For example, the image generation module 100 generates USD-Python and USD by inputting a prompt into a model for generating USD-Python.
[0119] The image generation system 1 replaces the USD file to be modified (step S111). For example, the image generation module 100 replaces the generated USD with the USD file to be modified. In this manner, the image generation system 1 executes a process to update the USD file. For example, the image generation system 1 executes a process to update all of the multiple data sets of the USD file. Then, the image generation system 1 executes the process of step S108. For example, the image generation module 100 executes a rendering process using the replaced, i.e., modified, USD file.
[0120] Here, a user interface (UI) related to the above-mentioned correction will be described. As shown in FIG. 20, the video generation system 1 provides the user with the content CT31. FIG. 20 is a diagram showing an example of the user interface. The content CT31 is a display screen (content) for presenting information related to the generated video, such as a storyboard, and for accepting correction instructions from the user. For example, the client UI display unit 400 displays the content CT31.
[0121] The user inputs information instructing corrections to the generated video via the content CT 31. For example, when the user clicks on a specific part of the video and inputs corrections in text, the video generation system 1 updates the USD and updates (changes) the video to reflect the corrections.
[0122] 20 shows a case where the user selects the top cut (thumbnail) image and instructs a correction to "child skips and runs to mother." In this case, the video production system 1 updates the USD of the part of the generated video that corresponds to the top cut (thumbnail) image based on the correction instruction to "child skips and runs to mother," and generates a video that reflects the correction content.
[0123] The above UI is merely an example, and the image generation system 1 may accept the user's correction instruction in various ways. For example, the image generation system 1 may provide the user with a content CT32 and accept the user's correction instruction as shown in FIG. 21. FIG. 21 is a diagram showing an example of a user interface. The content CT32 is a display screen (content) for accepting the user's correction instruction. For example, the client UI display unit 400 displays the content CT32.
[0124] Furthermore, the video generation system 1 may provide the user with a content CT33 and accept a correction instruction from the user, as shown in Fig. 22. Fig. 22 is a diagram showing an example of a user interface. The content CT33 is a display screen (content) for accepting a correction instruction from the user. For example, the client UI display unit 400 displays the content CT33.
[0125] The user inputs information instructing corrections to the generated video via the content CT32 or the content CT33. In this way, the video generation system 1 accepts the correction instructions from the user, updates the USD based on the correction instructions, and generates a video reflecting the correction contents. For example, the user can select a person or object that appears in the video, and edit the motion settings for the selected target by text. The user can also change the assets of the selected target. The user can also edit the camera work by text. In addition to the above, the user can also edit the background and lighting settings of the video by text.
[0126] Although the above description has been given as an example of a moving image, the video production system 1 may also perform correction processing on sound information (BGM, SE, narration, dialogue, etc.), text logo information, etc., based on a user's correction instruction.
[0127] <1-4-5. Evaluation example> Furthermore, the video generation system 1 executes an evaluation process. For example, the video generation system 1 generates an evaluation text for the scenario data using an AI model such as model M10. For example, the video generation system 1 may generate scenario data again based on an instruction for editing (correction, etc.) made by a user based on the evaluation text generated for the scenario data. The video generation system 1 presents the generated evaluation text. This allows the user to create a video while viewing the evaluation.
[0128] For example, the video generation system 1 presents an evaluation text about the scenario data to a user, and receives editing instructions for the scenario data from the user who has confirmed the presented evaluation text. The video generation system 1 generates scenario data based on the editing instructions received from the user. For example, the video generation system 1 changes (updates) the contents of the scenario data based on the editing instructions received from the user. The video generation system 1 generates code based on the scenario data generated based on the evaluation text. The video generation system 1 generates video data using the code generated based on the evaluation text.
[0129] The video generation system 1 may automatically generate (update) at least one of the scenario data or the code based on the evaluation text. For example, the video generation system 1 may change the content of the scenario data so that it corresponds to the content indicated by the evaluation text for the scenario data. For example, when the evaluation text for the scenario data indicates that the orientation of a certain character is not good, the video generation system 1 generates scenario data in which the orientation of the character is changed. Note that the above is merely an example, and the video generation system 1 may generate at least one of the scenario data or the code by appropriately using the evaluation text.
[0130] In the video generation system 1, after generating a video (video), scenario information, video information, sound information, text / logo information, etc. are stored as text, video file, sound file, image file, etc. Therefore, the video generation system 1 can perform evaluation processing using these information. The video generation system 1 modifies the 3DCG assets and rendering method that are the source of the existing video based on the user's input information, the text information of the generated storyboard, and the evaluation text, outputs a prompt for outputting a code that configures a new video, provides the prompt to a natural language model, outputs the code that configures the video, and generates a video. For example, the video generation system 1 may output a prompt for generating evaluation text (corresponding to the evaluation text information EV1 in FIG. 5) based on the user's input information and the text information of the generated storyboard, and provide the prompt to a natural language model to generate the evaluation text. For example, the video generation system 1 may generate an evaluation text indicating the evaluation of the scenario by generating a prompt for evaluation based on the scenario information and inputting it to an AI (LLM, etc.).
[0131] Furthermore, for evaluating the composition of a video, the video generation system 1 obtains a caption for the video by inputting one frame of the video into an AI (Contrastive Captioner Model, Image Captioning Model, etc.). Then, the video generation system 1 generates an evaluation text indicating an evaluation of the composition of the video by inputting the obtained caption and scenario text into an AI (LLM, etc.). For example, the video generation system 1 may evaluate whether the image is as per the caption by comparing the obtained caption and scenario text with an AI (LLM, etc.). Furthermore, the video generation system 1 may accept a specification of the evaluator type that the user desires to be the one to evaluate. For example, the video generation system 1 may perform evaluation from the perspective of a market strategy, a copywriter, a video director, a specific person, etc.
[0132] <1-4-6. Example of multiple cut selection> From here, some examples of the editing process will be described. In the conventional technology, there is a problem that multiple cuts cannot be simultaneously corrected on a storyboard. Thus, the conventional technology has a problem with usability, and there is room for improvement in usability. Therefore, the video production system 1 may select multiple cuts and perform the editing process (correction process) as shown in FIG. 23. FIG. 23 is a diagram showing an example of the editing process. As a specific example, FIG. 23 is a diagram showing an example of the editing process based on the selection of multiple cuts.
[0133] 23, the video production system 1 may provide the user with content CT41 and accept a correction instruction according to the user's selection of multiple cuts. The content CT41 is a display screen (content) for accepting a user's correction instruction for a video including multiple cuts (scenes) such as cuts CU1 to CU4. For example, the client UI display unit 400 displays the content CT41.
[0134] The user selects multiple cuts from among the cuts CU1 to CU4 via the content CT41, and inputs information instructing corrections to the selected multiple cuts. For example, the user selects cuts CU1 to CU3 by performing an operation to select the range in which cuts CU1 to CU3 are displayed (such as an operation to surround the range) from among the cuts CU1 to CU4 in the content CT41, or by clicking each of the cuts CU1 to CU3.
[0135] Then, after selecting cuts CU1 to CU3, the user inputs a prompt (text information, etc.) indicating a correction instruction as user input information, thereby instructing the video production system 1 to correct the cuts CU1 to CU3. The video production system 1 executes correction for the cuts CU1 to CU3 in response to the correction instruction from the user. This allows the user to select multiple cuts and correct the configuration and contents of the selected cuts by inputting a prompt. For example, the scenario information for each cut includes the contents of the cut, the characters, the shooting time of the cut, etc., and the video production system 1 regenerates the scenario by providing the user's input information and scenario information to the AI (such as LLM). Note that the video production system 1 executes any correction process based on the contents of the user's correction instruction. For example, the video production system 1 may update the USD file of the video as necessary, or may change only the order of the cuts while leaving the USD file as it is. The above-mentioned process allows the video production system 1 to improve usability.
[0136] <1-4-7. Example of using time information> In the conventional technology, there is a problem that after a specific cut is corrected, other scenes affected by the correction are not corrected (changed). Thus, the conventional technology has a problem with usability, and there is room for improvement in usability. Therefore, the video production system 1 may perform an editing process using time information, as shown in FIG. 24. FIG. 24 is a diagram showing an example of the editing process. As a specific example, FIG. 24 is a diagram showing an example of an editing process based on the correction content determined according to the time information. For example, the video production system 1 inserts date and time information (also called "time information" or "date information") generated by AI into each cut, and determines the cutting of the video and the correction content based on the information.
[0137] In FIG. 24, the video production system 1 manages each cut by associating it with date and time information (time information) corresponding to the cut. For example, time information TI11 indicating 10:31 on December 21, 2023 is associated with cut CU11. Also, time information TI12 indicating 10:41 on December 21, 2023 is associated with cut CU12. In this way, cuts CU11 and CU12 are cuts that are close in time (close to each other). In this case, when cut CU11 is modified, the video production system 1 also reflects the modification in cut CU12. Time information (date information) is associated with each cut of the video data.
[0138] For example, when the clothing of person X in cut CU11 is modified, the video production system 1 also modifies the clothing of person X in cut CU12 in the same way as the clothing of person X in cut CU11. For example, the video production system 1 compares time information between each cut, and when there is a cut (also called an "influenced cut") whose time difference with the modified cut (also called a "cut to be modified") is within a predetermined range, it determines that the modification based on the modification content in the cut to be modified should also be reflected in the influenced cut.
[0139] Then, the video production system 1 executes corrections to the affected cuts based on the correction contents of the correction target cut. Note that the corrections (changes) of the affected cuts based on the influence between the cuts may be performed not only on human assets but also on environmental assets (weather, lighting, props such as candles that change over time, etc.).
[0140] For example, cut CU21 is associated with time information TI21 indicating 10:31 on Dec. 21, 2023. Also, cut CU22 is associated with time information TI22 indicating 18:41 on Dec. 21, 2023. In this way, cuts CU21 and CU22 are cuts that are distant in time (separate). In this case, when cut CU21 is modified, the video production system 1 does not reflect the modification in cut CU22.
[0141] For example, when the clothing of person X in cut CU21 is modified, the video production system 1 does not modify the clothing of person X in cut CU22 in response to the modification of the clothing of person X in cut CU21. For example, the video production system 1 compares time information between each cut, and if there is no cut (affecting cut) whose time difference with the modified cut (cut to be modified) is within a predetermined range, it determines not to reflect the modification based on the modification content in the cut to be modified in other cuts.
[0142] For example, when generating a scenario, the video generation system 1 generates date and time information for each cut using AI (such as LLM) and saves it as meta information for each cut. This date and time information is used to consider the relationship with the previous and next cuts and to understand the season, etc. The generated fictitious date and time information is included to maintain the content of each cut and the relationship between each cut. For example, if the shot is a morning shot when it is snowing, the video generation system 1 may set it to 6:30 AM on January 24th. Also, for example, if the shot has a strong relationship with the previous and next cuts, the video generation system 1 may set it to a nearby time period on the same day, such as 6:30 AM on January 24th and 7 AM on January 24th.
[0143] For example, the video production system 1 changes the clothes worn by the user, the sun lighting settings, the haze of the air, etc. according to the season and time. Also, if the date and time information of a target cut and a previous cut are close, changing the target cut also affects the previous and next cuts. For example, if the same person appears in another cut and the time is close to that of the target cut, changing the clothes in the target cut also changes the clothes in the other cut. Through the above-mentioned processing, the video production system 1 can improve usability.
[0144] <1-4-8. Response examples according to the degree of verbalization> In the conventional technology, when the user cannot specifically verbalize the content of the correction, it is difficult to make corrections according to the user's intention. Thus, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the image generation system 1 may enable corrections according to the user's intention by the process described below, even when the user cannot specifically verbalize the content of the correction.
[0145] For example, the image production system 1 may change the reply method according to the input sentence. As shown in Fig. 25, the image production system 1 may vary the response according to the abstraction level and the degree of verbalization of the user's instruction. Fig. 25 is a diagram showing an example of an editing process. As a specific example, Fig. 25 is a diagram showing an example of an editing process according to the degree of verbalization.
[0146] Response example AP1 in Fig. 25 shows a case where a user issues a purpose-based correction instruction such as "Please make the video so that the user who watches it feels like everyday life". In this case, image generation system 1 generates an image with a specific instruction text attached. For example, in response to a purpose-based correction instruction such as "Please make the video so that the user who watches it feels like everyday life", image generation system 1 attaches a text such as "Lower your hands at a natural angle and move them in accordance with the direction of your body" and presents the video corrected based on the text to the user by displaying it on a display or the like.
[0147] Moreover, response example AP2 in FIG. 25 shows a case where the user instructs to correct an abstract sentence such as "Keep both arms in a natural position." In this case, the image generation system 1 presents a plurality of concrete sentences and allows the user to select one. For example, the image generation system 1 presents a plurality of sentences such as "Put your hands down and move them in the direction of the sky," "Scratch your head with your hands and then put your hands down," and "Put your hands in your pockets" to the user by displaying them on a display or the like. Then, the image generation system 1 presents to the user an image corrected based on the sentence selected by the user from the plurality of sentences by displaying them on a display or the like.
[0148] 25 shows a case where the user gives a specific instruction to correct a sentence, such as "Please look at the car in front of you on the right in 2 seconds." In this case, the image generation system 1 presents the image, which has been changed (corrected) according to the sentence entered by the user, to the user by displaying it on a display or the like. For example, the difference between a purpose-based sentence, an abstract sentence, and a specific sentence may be determined by an AI (such as an LLM), and the image generation system 1 may change the generated prompt accordingly and pass the processing to the AI (such as an LLM).
[0149] For example, the video production system 1 may allow a user to specify motion or camera movement using a sensor. As shown in Fig. 26, the video production system 1 may superimpose (overlap) motion information obtained from an arbitrary sensor such as a Web camera on the video and accept user corrections. Fig. 26 is a diagram showing an example of editing processing. As a specific example, Fig. 26 is a diagram showing an example of editing processing by superimposing display on the video.
[0150] For example, the video generation system 1 displays motion information MT obtained from any sensor such as a mobile motion capture or a web camera superimposed on the video MV11 and accepts a user's correction instruction. In this way, the video generation system 1 displays motion information including facial expressions superimposed on the current video, and accepts a user's correction instruction based on a visualized state of the difference between the current video and the motion information. For example, if it is a 4-second cut, the 4-second cut is always played back, and the user can change the motion information by himself or herself as many times as necessary until he or she likes it. For example, if there is motion information that the user likes, the video generation system 1 may finally generate natural motion information to change the motion of the video.
[0151] The user's motion is tracked using a mobile motion capture device, a web camera, etc. The user may also select in advance from the UI using the mouse which character's motion to modify. The size and position of the portion of the user's motion to be superimposed is determined based on the size and orientation of the head and body in the video. The user may adjust the size, position, and orientation in detail using the mouse and keyboard. The motion may be recorded by repeating the cuts shown in the image above, or by starting recording several seconds after pressing the start button and getting into position. After filming, the user may press the motion generation button to apply the input motion to the specified character.
[0152] In addition, since the captured motion itself may be unnatural, the image generation system 1 may use a motion-to-motion AI to estimate or generate the captured motion, convert it into a more natural motion, and then adapt it to the motion of the characters. In addition, the image generation system 1 may estimate or generate the motion using the input motion and natural text. For example, after inputting the motion, the image generation system 1 may send the motion to an AI (such as an LLM or a text&motion to motion generation model) along with the natural text input by the user, such as "Make the movement lively like this."
[0153] Furthermore, the user may change the camerawork by moving his / her hand as if it were a camera. For example, the image production system 1 acquires the movement of the user's hand using a sensor such as a Web camera. As shown in FIG. 27, the image production system 1 presents (superimposed display) the changed camerawork as a frame superimposed on the image. FIG. 27 is a diagram showing an example of editing processing. As a specific example, FIG. 27 is a diagram showing an example of superimposed display of camerawork on the image.
[0154] For example, the video production system 1 displays a frame CW1 showing the pre-change camerawork and a frame CW2 showing the post-change camerawork by superimposing them on the video MV12. Then, when the user instructs the video production system 1 to use the post-change camerawork, the video production system 1 executes a rendering process based on the post-change camerawork. In this way, when the user approves the post-change camerawork, the video is rendered with the post-change camerawork.
[0155] The user may also specify the camerawork using an IMU or ImageSLAM of the user's mobile terminal (such as a smartphone). For example, the image generation system 1 may present the main character by placing it in real space (such as on a real desk) using AR (Augmented Reality). The image generation system 1 displays the changed camerawork by superimposing a frame on the image in the same manner as in the case shown in FIG. 27.
[0156] As described above, in the video generation system 1, the user may use the movement of his / her hand or smartphone to determine the motion and the movement of the camera. For example, when the user determines the camerawork by the movement of his / her hand, the user may imagine a person specified by the other hand and determine the relative position of the camera from the distance between the left hand (camera) and the right hand (person). In addition, the video generation system 1 displays the camerawork changed by the user's input of the hand or camera as a frame superimposed on the video, and the user may then fine-tune the frame and its movement by operating the mouse. In addition, since the input camerawork is input by hand, it may be unnatural, so the video generation system 1 may correct the camerawork to a natural one using a motion to motion generation model, a text&motion to motion generation model, or the like. Through the above-mentioned processing, the video generation system 1 can improve usability.
[0157] <1-4-9. Examples of using qualitative values> In the conventional technology, there are cases where consideration is not given to how the system determines the current state and the change history. For example, there are cases where the user wants to make adjustments by comparing with the current state, such as "stand a little further back" or "make it a little brighter," and there is a problem with how the system should understand (grasp) the current state. Thus, the conventional technology has a problem with usability, and there is room for improvement in usability. Therefore, the video generation system 1 may be capable of appropriately determining the current state and the change history by the process shown below.
[0158] For example, the video production system 1 may store each point such as the speed of movement and brightness as a quantitative numerical value and correct the video by comparing the values. The video production system 1 may perform an editing process using qualitative values as shown in Fig. 28. Fig. 28 is a diagram showing an example of the editing process. As a specific example, Fig. 28 is a diagram showing an example of the editing process based on the correction content determined according to the qualitative values.
[0159] 28, value information VL21 indicates a quantitative value associated with video MV21. For example, value information VL21 indicates values associated with cuts of video MV21, including qualitative values such as the brightness of the entire town, people's walking speed, the speed at which people's faces move, people's positions, car speeds, and car positions. In this way, video production system 1 holds all values that can be changed as quantitative values and changes the video by comparing them with the values.
[0160] 28, the user gives a correction instruction for video MV21, "move your face a little slower," and video production system 1 generates video MV22 by slowing down the speed at which the face moves in video MV21. Based on the correction instruction, "move your face a little slower," video production system 1 generates video MV22 with value information VL22 by reducing the value of the speed at which the person's face moves, in the value information VL21 of video MV21, from "21" to "10."
[0161] 28, the value information VL22 indicates a quantitative value associated with the corrected video MV22. For example, the value information VL22 indicates a value associated with a cut of the video MV22, including qualitative values such as the brightness of the entire town, the walking speed of people, the speed at which people move their faces, the positions of people, the speed of cars, and the positions of cars, for example.
[0162] In this way, the image generation system 1 may perform image editing processing using qualitative values. For example, the image generation system 1 may store quantitative values for each point (item) such as movement speed and brightness, and compare the values to modify the image. As described above, classifications such as the brightness of the entire city and people's walking speed, for which qualitative values are stored, are items that are set in advance. For example, the image generation system 1 may set and modify each of these using AI (such as LLM) from natural language, or may directly modify the set values. Through the above-mentioned processing, the image generation system 1 can improve usability.
[0163] An example of a method for acquiring each value is shown below, but the acquisition method is not limited to the following and may be another acquisition method. For example, for lighting, the image generation system 1 acquires the setting values (position, rotation, strength, color, etc.) of the light in the 3DCG. For color grading, the image generation system 1 acquires the white balance, color temperature, color cast correction, saturation, exposure, contrast, highlight, shadow, white level, black level, color, LUT setting, etc. set in composite editing. For walking speed, the image generation system 1 acquires the position movement speed of the hip bone of the target 3D model. For face movement speed, the image generation system 1 acquires the rotation speed of the head bone. For position, the image generation system 1 acquires the position of the 3D model.
[0164] <1-4-10. Example of rendering process during input> The conventional technology has an issue that it is difficult for the user to wait for the rendering time (low usability). Thus, the conventional technology has an issue with usability, and there is room for improvement in usability. Therefore, the image generation system 1 may improve the usability regarding rendering by the process shown below.
[0165] For example, the image production system 1 may start rendering processing in the middle of text input. The image production system 1 may perform rendering processing in the middle of user input, as shown in Fig. 29. Fig. 29 is a diagram showing an example of editing processing. As a specific example, Fig. 29 is a diagram showing an example of editing processing based on rendering processing in the middle of input.
[0166] In FIG. 29, the video generation system 1 executes rendering processing and the like in response to the user's input of text TX31, "scratching his head with his hand," and displays video MV31. For example, the "|" at the end of text TX31 indicates that the user is in the middle of inputting a correction instruction. Once the video generation system 1 is able to understand the text as a sentence, it starts processing in the background. For example, the video generation system 1 starts rendering processing and the like in the background at the point in time when the user has input up to "scratching his head with his hand." In this way, the video generation system 1 may start rendering processing in the middle of the user's input of text.
[0167] In FIG. 29, the user changes the sentence from text TX31 to text TX32, "Put your hands down." For example, the "|" at the end of text TX32 indicates that the user is in the middle of inputting a correction instruction. The video generation system 1 executes processing in response to the change from text TX31 to text TX32. For example, when the sentence is changed, the video generation system 1 stops the processing being performed at that time and starts the processing again. For example, the video generation system 1 ends the processing that was being performed based on the text TX31, and executes rendering processing and the like based on the text TX32 to display the video MV32.
[0168] In Fig. 29, the user changes the text TX32 to text TX33, "Keep your hands down, in a natural way", completes the sentence, and instructs execution of processing by pressing a start button or the like. The video generation system 1 ends the background processing that has started, and displays video MV33 for which rendering processing and the like have been performed based on the text TX33. Through the above-mentioned processing, the video generation system 1 can improve usability.
[0169] <1-4-11. Example of confirmation work when editing> In the conventional technology, there is a problem that the usability is low in the confirmation work at the time of editing, and there is a room for improvement. Therefore, the video generation system 1 may perform the confirmation work at the time of editing as shown in Fig. 30. Fig. 30 is a diagram showing an example of the editing process. As a specific example, Fig. 30 is a diagram showing an example of the confirmation work at the time of editing.
[0170] A confirmation task CP1 in Fig. 30 shows an example of a confirmation task when it is necessary to check changes in the time axis direction, such as motion or camera work. For example, in confirmation task CP1, the user repeatedly selects a video that is close to ideal. Also, a confirmation task CP2 in Fig. 30 shows an example of a confirmation task other than when it is necessary to check changes in the time axis direction. For example, in confirmation task CP2, the user repeatedly selects an image that is close to ideal.
[0171] For example, the video generation system 1 may render images of all cuts and then render a video starting from the one to be edited. For example, the video generation system 1 may present the generated video to the user and allow the user to decide whether or not to edit the input information. For example, as shown in confirmation operations CP1, CP2, etc., when presenting the user with the generated results of several patterns of videos with variations in response to the user's input information, there is an issue that a waiting time is generated for rendering multiple videos. As such, the conventional technology has issues regarding usability, and there is room for improvement in usability.
[0172] Therefore, in order to reduce the waiting time, for tasks other than the confirmation task that requires viewing the change in the time axis direction (corresponding to confirmation task CP2), the video production system 1 renders only one or several frames in the video and presents candidates as images to the user. This enables the video production system 1 to reduce the rendering waiting time.
[0173] In addition, when the user edits the motion or camerawork, the video generation system 1 presents candidates using video. For example, methods for selecting a few frames from a video to be used for rendering include a method of simply rendering only the first and last two frames of the video, a method of rendering a few frames with a large amount of change in the animation in the USD file, and a method of having the AI select the frames to be displayed as highlights from among all the frames. For example, when the video generation system 1 renders multiple images, the user can check the generated results in the form of a flip book by hovering the mouse cursor over them.
[0174] Furthermore, the video generation system 1 can reduce waiting time during video preview by starting video generation for each pattern in sequence after completing image generation for candidate selection. As for the order of video generation for each pattern, in addition to a method of generating them in a random order, a method of allowing the user to select the rendering order by pressing a button on the UI, a method of rendering in order of the length of time the mouse cursor has been hovered over an image, and the like can be cited. Through the above-mentioned processing, the video generation system 1 can improve usability.
[0175] Here, an example of a processing flow related to the above-mentioned checking work will be described with reference to Fig. 31. Fig. 31 is a flowchart showing a processing procedure executed by the video production system. As a specific example, Fig. 31 is a flowchart showing a processing procedure related to editing processing.
[0176] 31, the video production system 1 generates several patterns of USD files based on user settings (step S201). If image writing of USD files for all patterns has been completed (step S202: Yes), the video production system 1 ends the process of writing images of files (e.g., steps S202 to S204).
[0177] If image writing of all patterns of USD files has not been completed (step S202: No), the video generation system 1 writes out images of the USD files for which writing has not been completed (step S203). The video generation system 1 displays the written images on the UI (step S204). After that, the video generation system 1 starts the process from step S205 onwards, and repeats the process of steps S202 to S204 until image writing of the files is completed.
[0178] If the video production system 1 has finished writing moving images of all patterns of USD files (step S205: Yes), it ends the process of writing moving images of files (for example, steps S205 to S207).
[0179] If the video writing of all patterns of USD files has not been completed (step S205: No), the video creation system 1 writes out the video of the USD files that have not been written out (step S206). The video creation system 1 displays the written out video on the UI (step S207). After that, the video creation system 1 repeats the processes of steps S205 to S207 until the video writing of the files is completed.
[0180] <1-4-12. Example of processing according to range selection> In the conventional technology, when a person (also called an "amateur") who has no experience (knowledge) in video (video) generation tries to create a video, it is difficult for him / her to judge what is good and what is bad, and he / she often makes an erroneous judgment. Thus, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the video generation system 1 may evaluate the generated video, as shown in FIG. 32. FIG. 32 is a diagram showing an example of evaluation processing according to range selection.
[0181] In Fig. 32, video generation system 1 performs evaluation processing in response to range selection. Video MV40 in Fig. 32 shows a moving image generated in response to user input. Video MV41 in Fig. 32 shows a state in which a user specifies a part of a person as the range they want evaluated, and text TX41 indicating the evaluation by video generation system 1 of that range is superimposed. In Fig. 32, video generation system 1 makes an evaluation (suggests a correction) for the part of the person specified by the user, as indicated by text TX41, that "it would be easier to convey emotions to the user if you showed this person's facial expression."
[0182] Video MV42 in Fig. 32 shows a state in which text TX42 indicating an evaluation by video production system 1 is further superimposed. In Fig. 32, video production system 1 performs an evaluation on the part of the person specified by the user, as indicated by text TX42, that "from the viewpoint of production, it would be better to capture the face from the front."
[0183] For example, if the user thinks the evaluation indicated by text TX42 is good, the video generation system 1 receives from the user an instruction for correction based on that evaluation. Then, the video generation system 1 generates and displays a plurality of candidate videos MV43, MV44, and MV45 corresponding to the evaluation indicated by text TX42. In this way, the video generation system 1 presents to the user a plurality of candidate videos MV43, MV44, and MV45 corresponding to the evaluation indicated by text TX42.
[0184] For example, a user specifies a range with a mouse, and image generation system 1 evaluates the specified range and starts a dialogue (discussion) with the user. In response to the range selected with the mouse, image generation system 1 starts an AI evaluation of that range. For example, image generation system 1 performs evaluation (presentation of correction suggestions) once every N seconds until the user instructs correction. When the user clicks with the mouse when they think it is good, image generation system 1 presents multiple correction suggestions based on the dialogue (discussion) up to that point. Through the above-mentioned processing, image generation system 1 can improve usability.
[0185] <1-4-13. Processing examples according to depth of field> In the conventional technology, when the number of polygons in the environmental asset is high or the texture resolution is high, and the asset is heavy, it is difficult to suppress the increase in the time required for rendering. Thus, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the video generation system 1 may perform processing according to the depth of field as shown in FIG. 33. FIG. 33 is a conceptual diagram showing an example of processing according to the depth of field.
[0186] In Fig. 33, the image generation system 1 uses a high polygon count and high resolution texture within a predetermined range in front and behind the subject, and uses a low polygon count and low resolution texture within a predetermined range in front and behind the subject, depending on the depth of field. In Fig. 33, objects with a high polygon count and high resolution texture (circles close to the subject) are shown with thick hatching, and objects with a low polygon count and low resolution texture (circles far from the subject) are shown with light hatching. In this way, the image generation system 1 changes the polygon count and texture resolution depending on the depth of field. The image generation system 1 calculates the depth of field using the following formulas (1) to (3).
[0187]
number
[0188]
number
[0189]
number
[0190] Equation (1) is a function for calculating the front depth of field (mm). The image generation system 1 calculates the front depth of field using equation (1). For example, in FIG. 33, the front depth of field corresponds to the front side of the subject (the side closer to the camera). Furthermore, equation (2) is a function for calculating the rear depth of field (mm). The image generation system 1 calculates the rear depth of field using equation (2). For example, in FIG. 33, the rear depth of field corresponds to the rear side of the subject (the side farther from the camera). Equation (3) is a function for calculating the depth of field. The image generation system 1 calculates the depth of field by adding together the front depth of field and the rear depth of field using equation (3).
[0191] For example, the image generation system 1 replaces parts closer to the camera than the forward depth of field and parts farther from the camera than the rear depth of field with parts with a lower number of polygons and texture resolution before rendering. For example, when creating a blur in terms of depth of field, the image generation system 1 reduces (lowers) the number of polygons and texture resolution. In this way, the image generation system 1 determines the number of polygons and texture resolution based on criteria such as focal length, F-number, and subject distance. For example, the image generation system 1 may make the above determination when generating a USD file. Through the above-mentioned processing, the image generation system 1 can improve usability.
[0192] <1-4-14. Example of playback processing during confirmation work> In the conventional technology, when the entire generated video is played back for confirmation, it is difficult to suppress an increase in the time required for the confirmation work. Thus, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the video generation system 1 may perform playback suitable for the confirmation work. The video generation system 1 may change the mode of playback in response to a user's operation. For example, the video generation system 1 may play back the video in a mode like a flip book in response to a user's operation.
[0193] The video production system 1 may advance the playback of the video by a predetermined number of seconds (e.g., 0.5 seconds) for each click or mouse wheel rotation by the user. For example, the video production system 1 advances frames within a cut according to the movement or position of the mouse or mouse wheel. The video production system 1 may advance the playback of the video by the number of seconds corresponding to the position of the mouse, as shown in FIG. 34.
[0194] In FIG. 34, the video generation system 1 may provide the user with content CT41 and accept an instruction of the degree to which the video is advanced (number of seconds, number of frames, etc.) by the user. The content CT41 is a display screen (content) for accepting the degree to which the user is to advance the video. The content CT41 arranges information for specifying the number of seconds to advance the video by superimposing it on the video. FIG. 34 shows a case where a value between 0 and 3.5 seconds can be specified, and the number of seconds to advance the video increases from left to right. For example, the client UI display unit 400 displays the content CT41.
[0195] The user inputs information specifying the number of seconds to advance the video when playing the video via the content CT41. In FIG. 34, the user operates the mouse to position the mouse cursor MS in the area marked "1.0" to specify the number of seconds to advance the video as 1.0 seconds. In this case, the video production system 1 advances the playback of the video at 1.0 second intervals in response to the user's specification of the number of seconds to advance the video. Note that the user may operate the mouse to position the mouse cursor MS in the area marked "1.0" and click to specify the number of seconds to advance the video as 1.0 seconds. Through the above-mentioned process, the video production system 1 can improve usability.
[0196] <1-4-15. Highlighting example> In the conventional technology, there are cases where it is difficult to understand the generated parts of the video that have been changed by editing or the like, and there is a problem that it is difficult to suppress an increase in the time required for the confirmation work. Thus, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the video generation system 1 may perform highlighting as shown in FIG. 35. FIG. 35 is a diagram showing an example of highlighting the changed parts. For example, the video generation system 1 highlights the previous generation result and the changed parts. In FIG. 35, a case where a woman in the video has been changed is explained as an example.
[0197] Video MV51 in Fig. 35 shows a first highlighting mode. Video MV51 shows a mode in which the changed parts are highlighted by darkening (reducing brightness, etc.) parts other than the changed parts. Video production system 1 generates video MV51 and displays video MV51 to highlight the parts that have been changed by editing, etc.
[0198] Moreover, video MV52 in Fig. 35 shows a second highlighting mode. Video MV52 shows a mode in which the changed parts are highlighted (by adding color, etc.). Video production system 1 generates video MV52 and displays video MV52 to highlight the parts that have been changed by editing, etc. In this way, videos MV51 and MV52 show a case in which parts of people are highlighted when they have been changed.
[0199] Video MV53 in Fig. 35 shows a third highlighting mode. Video MV53 shows a mode in which the changed portion is highlighted by showing the time when the change occurred on the seek bar. Video production system 1 generates video MV53 including a seek bar on which colored points HL531 and HL532 are placed at positions corresponding to the time when the change occurred, and displays video MV53 to highlight the portion that has been changed by editing or the like.
[0200] Video MV54 in Fig. 35 shows a fourth highlighting mode. Video MV54 shows a mode in which the changed portion is highlighted by indicating the time when the change occurred on the seek bar. Video production system 1 generates video MV54 including a seek bar on which a colored bar HL54 is positioned in a range corresponding to the time period when the change occurred, and displays video MV54 to highlight the portion that has been changed by editing or the like.
[0201] As described above, when a user sets the settings related to video generation and regenerates a video, the generated video needs to be checked. In a UI that presents multiple generation results, the user has to bear the burden of checking each video one by one. In this regard, in order to reduce the burden of checking the videos, the video generation system 1 presents the user with the difference from the result of the previous video generation. For example, the video generation system 1 presents the user with the difference from the result of the previous video generation by highlighting only the parts of the video that have been changed or by displaying the time when the change occurred in the video on a seek bar. This allows the user to check only the parts that have been changed, and the video generation system 1 can reduce the burden of the checking work. Through the above-mentioned process, the video generation system 1 can improve usability.
[0202] For example, methods for extracting changed parts in a video include a method for detecting differences from a generated USD file, and a method for detecting differences by comparing a rendered video with a previously rendered video frame by frame. For example, in a method for detecting differences from a generated USD file, taking advantage of the characteristic of generating USD files each time and performing video rendering, the video generation system 1 compares the contents of USD files generated before and after the user updates the settings related to video generation, and detects objects that have changed and the time at which the changes occurred. Also, for example, in a method for detecting differences by comparing a rendered video with a previously rendered video frame by frame, the video generation system 1 compares the videos generated before and after the user updates the settings related to video generation, and detects changed pixels and the time at which the changes occurred.
[0203] <1-4-16. Example of showing the relationship between cuts> In the conventional technology, when each cut is corrected, there is a problem that the relationship between the previous and next cuts may become unclear. Thus, the conventional technology has a problem with usability, and there is room for improvement in usability. Therefore, the video production system 1 may present the relationship between cuts as shown in Fig. 36. Fig. 36 is a diagram showing an example of the presentation of the relationship between cuts.
[0204] 36 shows a case where cut CU61 is a modified cut (also called a "target cut"). A previous relationship bar TR60, which indicates that the cut is a cut that precedes the target cut, is superimposed on cut CU60 that precedes cut CU61, which is the target cut. For example, the previous relationship bar TR60 is a triangle that has its base on the right and extends leftward. The previous relationship bar TR60 is displayed in such a manner that it extends further to the left the further away it is from the target cut in time.
[0205] Also, a cut CU62 that is later than the target cut CU61 has a subsequent relationship bar TR62 superimposed thereon, indicating that the cut is later than the target cut. For example, the subsequent relationship bar TR62 is a triangle extending to the right with its base on the left side. A subsequent relationship bar TR63 that is further later than the cut CU62 that is later than the target cut CU61 has a subsequent relationship bar TR63 superimposed thereon, indicating that the cut is later than the target cut. For example, the subsequent relationship bar TR63 is a triangle extending to the right with its base on the left side.
[0206] The subsequent relationship bars TR62 and TR63 are displayed in such a manner that the further away they are from the target cut, the further to the left they extend. In Fig. 36, cut CU63 is later than cut CU62, so the subsequent relationship bar TR63 is displayed in such a manner that it extends further to the right than the subsequent relationship bar TR62. Note that the triangular display mode is merely one example of a display mode, and any display mode can be adopted as long as it is possible to present the temporal relationship and its amount.
[0207] The video production system 1 plays a video including cuts CU60 to CU63 in response to a user's operation. For example, when displaying cut CU60 in response to a user's operation, the video production system 1 superimposes a previous relationship bar TR60. For example, when displaying cut CU62 in response to a user's operation, the video production system 1 superimposes a next relationship bar TR62. For example, when displaying cut CU63 in response to a user's operation, the video production system 1 superimposes a next relationship bar TR63. In this way, when playing back cuts before and after a target cut, the video production system 1 presents the amount of distance from the target cut. In this way, when playing back cuts before and after the target cut, the video production system 1 allows the user to recognize whether the cut is before or after the target cut, and how far it is from the target cut, by superimposing information indicating the amount and direction on the screen according to the number of seconds away from the target cut. By the above-mentioned process, the video production system 1 can improve usability.
[0208] <1-4-17. Example of object selection> The conventional technology has an issue that it is difficult for the user to select an object during editing (low usability). Thus, the conventional technology has an issue regarding usability, and there is room for improvement in usability. Therefore, the video generation system 1 may select an object as shown in Fig. 37. Fig. 37 is a diagram showing an example of object selection.
[0209] In the video MV71 of Fig. 37, the user operates the mouse to select the right-hand person among the three people by positioning the mouse cursor MS71 in a range that includes the right-hand person. In this case, the video production system 1 accepts an operation to select the right-hand person where the mouse cursor MS71 is positioned as the target object. For example, the video production system 1 identifies the segmentation (range) corresponding to the right-hand person where the mouse cursor MS71 is positioned as the range selected by the user.
[0210] In the video MV72 in Fig. 37, the user operates the mouse to select the central person among the three people by positioning the mouse cursor MS72 in a range that includes the central person. In this case, the video production system 1 accepts an operation to select the central person where the mouse cursor MS72 is positioned as the target object. For example, the video production system 1 identifies the segmentation (range) corresponding to the central person where the mouse cursor MS72 is positioned as the range selected by the user.
[0211] In video MV73 in Fig. 37, the user operates the mouse to select the left person among the three people by positioning the mouse cursor MS73 in a range that includes the left person. In this case, video production system 1 accepts an operation to select the left person where the mouse cursor MS73 is positioned as the target object. For example, video production system 1 identifies the segmentation (range) corresponding to the left person where the mouse cursor MS73 is positioned as the range selected by the user.
[0212] In this way, the video generation system 1 recognizes the selection of the target object within the range recognized by segmentation. This allows the user to select the object he or she wants to specify by simply clicking it. For example, when the user changes the motion or assets of an object displayed in a video, the object to be changed must be selected. If the user can directly click on the object in the video, rather than using a UI that allows the user to select the object to be changed from a list of objects displayed in the video, the user can more intuitively select the object to be changed. The click range of an object in a video can be realized by segmenting the pixel area of each object for each frame of the video.
[0213] The click range of an object may be calculated in the following manner. For example, since object and camera information for each frame of a video is stored in USD, it is possible to map which object a pixel seen from the camera in that frame points to, and the image generation system 1 can then reverse-calculate the coordinates where the user clicked on the video and the object to be edited.
[0214] A method for determining the click range of an object includes a method of linking a person selected by segmenting an object for each pixel calculated backwards from the camera for each frame of a video with a target USD. For example, the video itself may include information about what is located in this area (x, y coordinates, etc.). For example, the video generation system 1 may select an object as shown in FIG. 38.
[0215] Frame FR in FIG. 38 indicates one frame of a video rendered from a camera. USD data OB in FIG. 38 indicates a 3D object and a camera on the USD. Because information on the 3D object and the camera is included in the USD, the video generation system 1 can map which location on the 3D object a pixel viewed from a camera (such as camera CM in FIG. 38) points to. This enables the video generation system 1 to map the coordinates clicked by the user on the video to the object. Through the above-mentioned processing, the video generation system 1 can improve usability.
[0216] <1-4-18. Example of using 3D model> The above-described process is merely an example, and the image generation system 1 may execute various processes other than the above-described process. In this regard, several embodiments will be described below.
[0217] For example, the video generation system 1 may perform a process of importing characters and props (such as products). As a method of importing characters and props (such as products) in the video generation system 1, for example, a 3D model or a three-dimensional drawing that is specified and desired to appear in the video may be registered and used in the video. For example, the video generation system 1 may register a 3D model input by a user and use it in the video. For example, the video generation system 1 may be capable of inserting photos of a product taken at various angles. In this case, the video generation system 1 may use a technology such as NeRF (Neural Radiance Fields) to generate a 3D model of the product from photos of the product taken at various angles and use it in the video.
[0218] For example, the video generation system 1 accepts a user's operation to specify an object to be changed from among video data, and generates code in which the 3D data of the object specified by the user's operation has been changed. For example, when a character in a video (also called "character A") is specified by the user as the object to be changed, and the user selects another character (also called "character B") from a registered 3D model as the changed character, the video generation system 1 generates code in which the 3D data of character A in the video has been changed to the 3D data of character B. This enables the video generation system 1 to generate video data in which character A in the video has been changed to character B indicated by the registered 3D model.
[0219] The above-mentioned process is merely an example, and the image generation system 1 may generate a code in which the 3D data of the object indicated by the user's operation is changed by any process. For example, the image generation system 1 may generate a code that has been changed by making a change to the 3D data itself of the object indicated by the user's operation in the video data. For example, the image generation system 1 may generate a code that has been changed by making a change to the external shape (height, etc.) of the 3D data of the object indicated by the user's operation in the video data. Furthermore, the image generation system 1 may not need to partially perform refiner processing after rendering for registered people and props. Furthermore, the image generation system 1 may prepare a marketplace within the image generation service and sell characters and props.
[0220] <1-4-19. Examples of using reference data> Furthermore, the video production system 1 may refer to videos and stories from a previously created (video) project. When a sequel to a previously created project is to be used, the video production system 1 may receive an input of the previously created project as reference data, as shown in Fig. 39. Fig. 39 is a diagram showing an example of processing using reference data.
[0221] For example, the video production system 1 acquires user input information regarding reference to other projects that the user inputs to the content CT 51. The content CT 51 is content for receiving user input information regarding an item for designating with a check mark whether or not to refer to other projects, an item for designating with a check mark the projects to refer to, an item for designating with a check mark which information of the referred projects to refer to, etc.
[0222] For example, the client UI display unit 400 displays the content CT51, and the sensor unit 300 accepts user input information. In Fig. 39, the user checks "Refer to other projects" to select to refer to other projects. The user also checks "Product X CM video" to select to refer to the CM video project for Product X.
[0223] The user also checks "Characters" and "Visual Style", choosing to refer to the characters and visual style of the commercial video project for Product X. The user also does not check "Story", "Content Style", or "BGM", choosing not to refer to the story, content style, or background music of the commercial video project for Product X.
[0224] The video production system 1 generates a prompt PT51, which is scenario generation information (first input information), using user input information accepted by the content CT51. Note that, although not illustrated in Fig. 39, the video production system 1 may generate the prompt PT51 using template input information such as the template TP1 shown in Fig. 6.
[0225] For example, the video production system 1 generates prompt PT51 by reflecting the characters and visual style in a commercial video project for product X. In FIG. 39, the video production system 1 generates prompt PT51 including a constraint specifying to use "Mike", a character in the commercial video for product X, and to use the visual style "cinematic" in the commercial video for product X. In this way, the video production system 1 generates a prompt for generating a scenario based on reference data of past projects in response to a user's selection. This allows the video production system 1 to generate video data for character A in the video based on a past project specified by the user.
[0226] <1-4-20. Examples of advantages of having 3D data> Since the image generation system 1 has 3D data internally, it has the following functions and advantages. The image generation system 1 has the following functions and advantages regarding the images (moving images) it generates. For example, the image generation system 1 can create more natural images through physical simulation. For example, the image generation system 1 can place an object on a platform, bounce it, roll it, and reproduce the natural swaying of cloth.
[0227] For example, the image generation system 1 can realistically reproduce the reflection of light according to the lighting and object material. When it is a bonnet or a mirror that is highly reflective, people or objects in front of it will be brighter due to the reflected light. When it is a cloth that is not very reflective, people or objects in front of it will not receive much reflected light.
[0228] For example, the video generation system 1 can fix lighting and objects, reducing the possibility of time-series breakdowns. For example, the video generation system 1 can express images without breakdowns even if the lighting position is changed during the video. For example, the video generation system 1 can output images in real time by setting a simple light source. In this case, the video generation system 1 does not need to perform refinement processing.
[0229] For example, the video generation system 1 can faithfully reproduce the product itself in the video by inputting product data and characters as 3D data. For example, the video generation system 1 can determine the depth, global position, normal, etc., so it can reduce the possibility of failure during refinery processing. For example, the video generation system 1 can fix the 3D position of a speaker or other sound source, making it easier to create interactions with sound. For example, the video generation system 1 can reduce rendering time by performing lighting processing such as Differential Rendering.
[0230] Furthermore, the image production system 1 has the following functions and advantages for editing (correcting) images (videos). For example, the image production system 1 can maintain the consistency of the surrounding environment and lighting, even when the camera position, angle, and camerawork are significantly corrected. For example, the image production system 1 can specify the position and angle three-dimensionally within a video.
[0231] For example, even after a certain amount of video has been generated, the video generation system 1 can change only specific things, such as the characters or props, and keep the rest of the video the same. For example, the video generation system 1 can change a character from a realistic human-like character to a two-headed character. For example, the video generation system 1 can change an installed signboard from a blackboard type to a plastic board. For example, the video generation system 1 can change only a part of the logo on a product package.
[0232] For example, the image generation system 1 can correct the image of a part that is not shown in the video frame by viewing the captured scene in 3D. For example, the image generation system 1 can place a light source in a part that is not shown. For example, the image generation system 1 can place a person or object in a part that is not shown and display only a shadow in the video frame. For example, the image generation system 1 can place a person or object in a part that is not shown and specify that a person who is shown looks into the eyes of a person who is not shown.
[0233] In addition to the above, the image generation system 1 has the following functions and advantages: For example, the image generation system 1 can easily convert content into 3D device content such as AR, VR (Virtual Reality), SRD (Spatial Reality Display), and 3D display.
[0234] <1-5. Example of processing flow from the user's perspective> Next, as an example of a processing flow seen from the user's perspective, a procedure of information processing by the image generation system 1 in response to a user operation will be described with reference to Fig. 40. Fig. 40 is a flowchart showing the flow of processing in response to a user operation.
[0235] 40, in the video production system 1, the user presses a project creation button (step S1). Then, in the video production system 1, the user inputs information required for video generation (step S2). For example, the user inputs information including the purpose of video production, the message to be conveyed through the video, the target users, the functional features of the product / service to be conveyed, the length of the video, the aspect ratio, etc.
[0236] Then, in the image production system 1, the user presses the storyboard, video, sound, and text logo creation buttons (step S3). For example, the image production system 1 may generate everything at once (for example, up to video data) in response to user operation, or may present a storyboard and allow the user to make some modifications in response to instructions from the user before generating the video, sound, and text logo.
[0237] Then, the image production system 1 modifies the storyboard, the video, the sound, and the text logo in response to the user's operation (step S4). For example, the image production system 1 may accept modifications to each of the items in the order of the user's preference.
[0238] Then, the video production system 1 performs export in response to a user operation (step S5). For example, the video production system 1 may perform export in a video file format such as mp4, avi, or mov in response to a user operation. Also, for example, the video production system 1 may perform export in a file format of any video editing software such as Premiere Pro, After Effects, or Davinci Resolve.
[0239] <1-6. About AI models> Each of the AI models used in the above-mentioned processes is not limited to the examples described in each section, and any internal structure can be adopted as long as it can output desired information in response to input. Any combination of input, output, and internal structure of the AI model can be adopted as long as it can output desired information.
[0240] The input of the AI model may be text, images, audio, 3D data, etc., or a combination thereof. The output of the AI model may be text, images, audio, 3D data, etc. Note that the above-mentioned inputs and outputs are merely examples, and the above-mentioned AI model may have any input and output.
[0241] In addition, the internal structure of the AI model can be any structure depending on the combination of input and output. In other words, the internal structure of the AI model can be any structure as long as it can produce a desired output for the input.
[0242] For example, the AI model may have a structure related to Transformer. For example, the AI model may have a structure related to Transformer and perform processing taking into account context such as context within data such as text and time-series data. For example, the AI model may have a self-attention mechanism. For example, the AI model may have any attention mechanism such as Single-Head Attention and Multi-head Attention. Note that the AI model does not need to have an attention mechanism.
[0243] The AI model may have a mechanism for extracting features from an input. For example, the AI model may have an encoder. The AI model may have a mechanism for generating information based on the extracted features. For example, the AI model may have a decoder.
[0244] The AI model may have a structure related to a CNN (Convolutional Neural Network). For example, when the AI model performs processing on an image, the AI model may have a structure related to a CNN. For example, the AI model may have at least one of a convolution layer, a pooling layer, a fully connected layer, and the like.
[0245] The above-described internal structure is merely an example, and the above-described AI model may have any internal structure. For example, the AI model may have a skip connection. The AI model may also have a structure related to a diffusion model.
[0246] In addition, the above-mentioned AI model may be generated (learned) by any learning process. The AI model may be a machine learning model learned using any machine learning method. For example, the AI model may be a model generated based on a so-called Foundation Model by fine-tuning the foundation model so as to apply the foundation model to a specific task (e.g., scenario data generation, code generation, etc.). For example, an AI model such as the above-mentioned LLM may be a model generated by fine-tuning the foundation model so as to apply the foundation model to a specific task.
[0247] The base model here is a model that has been trained to be applicable to various tasks, for example to be able to perform a wide variety of tasks. For example, the base model is a neural network that has been pre-trained with a large amount of unlabeled data set. The base model may have any structure, such as a Transformer-based architecture. For example, the base model is generated by self-supervised learning using data without a correct answer label. As described above, the base model is fine-tuned to be adaptable to a wide range of downstream tasks.
[0248] For example, when applied to a task of scenario data generation, the base model is fine-tuned so that it can be adapted to the task of scenario data generation, and an AI model (model M1, etc.) applied to the task of scenario data generation is generated. Also, when applied to a task of code generation, the base model is fine-tuned so that it can be adapted to the task of code generation, and an AI model (model M3, etc.) applied to the task of code generation is generated. Also, when applied to a task of sound data generation, the base model is fine-tuned so that it can be adapted to the task of sound data generation, and an AI model (model M4, etc.) applied to the task of sound data generation is generated. Also, when applied to a task of text logo generation, the base model is fine-tuned so that it can be adapted to the task of text logo generation, and an AI model (model M5, etc.) applied to the task of text logo generation is generated.
[0249] For example, an AI model (such as model M1) applied to a task of generating scenario data is trained using training data including a combination of input information corresponding to the AI model and scenario data (also called "correct answer information") that is the correct output when the input information is input. The training data, such as the input information and the correct answer information, may be data created by a person, or may be data automatically generated by a computer that generates the training data. For example, the scenario data that is the correct answer information may be data created by a person. Below, model M1 will be briefly described as an example. For example, model M1 is trained to output correct answer information corresponding to each input information when each piece of input information in the training data is input. For example, model M1 is trained by adjusting (correcting) parameters (connection coefficients) using a method such as backpropagation (error backpropagation method) so that the error between the output of model M1 when certain input information is input and the correct answer information corresponding to the input information is reduced. In addition, other AI models such as an AI model applied to a code generation task (e.g., model M3), an AI model applied to a sound data generation task (e.g., model M4), an AI model applied to a text logo generation task (e.g., model M5), an AI model applied to an evaluation task (e.g., model M10), and an AI model applied to an image improvement processing task (e.g., model M11), may also be trained using a similar learning process.
[0250] The above-mentioned learning process is merely an example, and the above-mentioned AI model is learned by any learning process according to the input, output, and internal structure of the AI model. For example, the AI model may be learned by an unsupervised learning method such as GAN (Generative Adversarial Network). The AI model may also be learned in a distributed state without aggregating data such as federated learning. In this case, a local model collected by each video generation service device (server, etc.) may be generated in the service, and a server (aggregation server) that aggregates information (parameters, etc.) of the local models generated by each video generation service device (server, etc.) may generate a global model using the information of the local model. In this case, the video generation system 1 may receive the global model generated by the aggregation server from the aggregation server, and use the received global model as an AI model for processing.
[0251] In this way, the above-mentioned AI models may be generated (learned) by any computer. That is, the learning process for generating the AI models may be performed by any device (computer, etc.) of the video production system 1, or may be performed by a device outside the video production system 1. For example, when a device outside the video production system 1 generates at least one of the above-mentioned AI models, the video production system 1 acquires the AI model from the device outside the video production system 1 and performs processing using the acquired AI model.
[0252] <2. Other embodiments> The processing according to each of the above-described embodiments may be implemented in various different forms (variations) other than the above-described embodiments and variations.
[0253] <2-1. Other configuration examples> The above-mentioned configuration of the image generation system 1 is merely an example, and any manner of division of functions in the image generation system 1 can be adopted. That is, the above-mentioned configuration is merely an example, and the image generation system 1 may have any manner of division of functions and any configuration as long as it can provide the above-mentioned service related to image generation. For example, the image generation system 1 may be configured by one device (such as a computer) that performs the above-mentioned processing. In this case, one device of the image generation system 1 may have the functions of the image generation module 100, the information acquisition module 200, the sensor unit 300, and the client UI display unit 400. For example, the image generation service provided by the image generation system 1 may be provided to the user as a program such as a tool (AI Assist Creation Tool) that runs on a terminal device (such as the computer 20) used by the user.
[0254] <2-2.Other> Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.
[0255] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.
[0256] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.
[0257] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.
[0258] <3. Effects of the present disclosure> As described above, the video generation system (video generation system 1 in the embodiment) according to the present disclosure includes an acquisition unit (input text acquisition unit 210 in the embodiment), a scenario generation unit (scenario-oriented generation unit 131 in the embodiment), a code generation unit (video-oriented generation unit 132 in the embodiment), and a video data acquisition unit (video generation unit 140 in the embodiment). The acquisition unit acquires an input query related to video generation from a user. The scenario generation unit generates scenario data related to video generation based on the input query. The code generation unit generates code for constructing 3D data based on the scenario data. The video data acquisition unit acquires video data based on the code.
[0259] In this way, the video generation system of the present disclosure generates code for constructing 3D data based on scenario data generated based on an input query from a user, and obtains video data based on the code, thereby enabling the video generation system to obtain video data in response to an input query from a user.
[0260] The video production system also includes an image quality improvement unit (in the embodiment, an image refinement unit 143). The image quality improvement unit executes an image quality improvement process that improves the image quality of the video data. In this manner, the video production system can obtain high-quality video by improving the image quality of the video data.
[0261] The image quality improvement unit improves the image quality of the video data by performing an image quality improvement process based on the text prompt. In this way, the video production system can obtain high-quality video by improving the image quality of the video data based on the text prompt.
[0262] The image quality improvement unit also performs the image quality improvement process based on a text prompt that specifies the target of the image quality improvement process in the video data. In this way, the video production system can obtain a high-quality video by improving the image quality of the video data for the specified target.
[0263] In addition, the image quality improvement unit executes the image quality improvement process based on a text prompt that specifies the target determined to require improvement. In this way, the image production system can obtain high-quality video by improving the image quality of the video data for the target determined to require improvement.
[0264] The video generation system also includes a display control unit (in this embodiment, a client UI display unit 400). The display control unit displays a storyboard based on the scenario data. In this manner, the video generation system displays a storyboard based on the scenario data, thereby making it possible to present information in a manner that is highly convenient for the user.
[0265] Moreover, the storyboard is configured to display video data for each cut of the video. In this manner, the video production system is able to present information in a manner that is highly convenient for the user by configuring the storyboard to display video data for each cut of the video.
[0266] The video production system also includes a sound production unit (sound production unit 150 in this embodiment). The sound production unit generates sound data corresponding to the video data based on the scenario data and the video data. In this way, the video production system can obtain a video including sound by generating sound data corresponding to the video data.
[0267] The video production system also includes a text generation unit (text / logo generation unit 160 in this embodiment). The text generation unit generates text data indicating text to be displayed on a video displayed by video data, based on the scenario data. In this way, the video production system can obtain a video including text by generating text data corresponding to the video data.
[0268] The video production system also includes a logo generation unit (text / logo generation unit 160 in this embodiment). The logo generation unit generates logo data indicating a logo to be displayed on a video displayed by the video data, based on the scenario data. In this way, the video production system can obtain a video including a logo by generating logo data corresponding to the video data.
[0269] In addition, the input query includes at least one of text, image, audio, and 3D data. In this manner, the video generation system can acquire video data in response to the input query from the user by the input query including at least one of text, image, audio, and 3D data.
[0270] The video generation system also includes a first output unit (scenario-oriented generation unit 131 in this embodiment). The first output unit outputs scenario generation information used by the scenario generation unit to generate scenario data based on the input query. The scenario generation unit generates scenario data based on the scenario generation information. In this way, the video generation system can acquire video data in response to the input query from the user by generating scenario data based on the scenario generation information generated based on the input query.
[0271] Furthermore, the first output unit generates a first prompt for generating scenario data based on the input query as scenario generation information. The scenario generation unit generates the scenario data based on the first prompt. In this manner, the video generation system can acquire video data in response to the input query from the user by generating scenario data based on the first prompt.
[0272] Furthermore, the first output unit generates, based on the input query, first input information used as an input of a first model for generating scenario data as scenario generation information. The scenario generation unit inputs the first input information generated using the input query to the first model and causes the first model to output scenario data, thereby generating scenario data. In this manner, the video generation system can acquire video data in response to an input query from a user by generating scenario data using the first model.
[0273] The video generation system also includes a second output unit (video generation unit 132 in this embodiment). The second output unit outputs code generation information used by the code generation unit to generate code for constructing 3D data based on the scenario data. The code generation unit generates code based on the code generation information. In this way, the video generation system can acquire video data in response to an input query from a user by generating code based on the code generation information generated based on the scenario data.
[0274] Further, the second output unit generates, as code generation information, a second prompt for outputting a code for configuring the 3D data based on the scenario data. The scenario generation unit generates the code based on the second prompt. In this manner, the video generation system can acquire video data in response to an input query from a user by generating the code based on the second prompt.
[0275] Furthermore, the second output unit generates, based on the input query, second input information used as an input of a second model for generating code as information for code generation. The scenario generation unit inputs the second input information generated using the scenario data to the second model and causes the second model to output code, thereby generating code. In this way, the video generation system can acquire video data in response to an input query from a user by generating code using the second model.
[0276] The video production system also includes a reception unit (sensor unit 300 in this embodiment). The reception unit receives operations related to video editing from a user. The code generation unit generates code through editing based on the operations. In this way, the video production system can appropriately acquire video corresponding to the user's editing by generating code in response to the video editing operations from the user.
[0277] The receiving unit also receives an operation specifying a motion or camera movement using the sensor. The code generating unit generates a code corresponding to the motion or camera movement indicated by the operation. In this manner, the video generating system can appropriately acquire a video corresponding to the user's editing by generating a code in response to a user's operation specifying a motion or camera movement.
[0278] The receiving unit also receives an operation to select multiple cuts from the video data. The code generating unit generates a code in which the portions corresponding to the multiple cuts indicated by the operation have been changed. In this manner, the video generating system can appropriately acquire a video corresponding to the user's editing by generating a code in response to the user's selection of multiple cuts.
[0279] Furthermore, date information is associated with each cut of the video data. The code generation unit determines the content of the edit indicated by the operation based on the date information of each cut of the video data. In this way, the video generation system can appropriately acquire a video corresponding to the user's editing by determining the content of the edit based on the date information of each cut of the video data.
[0280] The receiving unit also receives an operation to specify an object to be changed from among the video data. The code generating unit generates a code in which the 3D data of the object specified by the operation has been changed. In this manner, the video generating system can appropriately acquire a video corresponding to the user's editing by generating a code in which the 3D data specified by the user as the object to be changed has been changed.
[0281] The video production system also includes an evaluation unit (evaluation unit 180 in this embodiment). The evaluation unit generates information indicating an evaluation of at least one of the scenario data and the video data. In this manner, the video production system can evaluate the generated information by generating information indicating an evaluation of at least one of the scenario data and the video data.
[0282] Furthermore, the code generator generates the code based on the evaluation. In this manner, the video generation system generates the code based on the evaluation, thereby enabling the video generation system to appropriately acquire information in accordance with the evaluation.
[0283] Furthermore, the scenario generation unit generates scenario data based on the evaluation. In this manner, the image generation system generates scenario data based on the evaluation, thereby enabling the image generation system to appropriately acquire information in accordance with the evaluation.
[0284] Furthermore, the scenario generation unit generates a code based on the scenario data generated based on the evaluation. In this manner, the video generation system generates a code based on the scenario data generated based on the evaluation, thereby enabling the video generation system to appropriately acquire information according to the evaluation.
[0285] <4. Hardware configuration> An information processing device (information equipment) having the image generation module 100, the information acquisition module 200, the client UI display unit 400, and the like according to each of the above-described embodiments is realized by a computer 1000 having a configuration as shown in FIG. 41, for example. FIG. 41 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device. The image generation module 100 according to the embodiment will be described below as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.
[0286] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.
[0287] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.
[0288] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an image generation program according to the present disclosure, which is an example of program data 1450.
[0289] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.
[0290] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.
[0291] For example, when computer 1000 functions as image generation module 100 according to the embodiment, CPU 1100 of computer 1000 executes an image generation program loaded onto RAM 1200 to realize functions of control unit 1301, etc. Also, HDD 1400 stores the image generation program according to the present disclosure and data in storage unit 1302. Note that CPU 1100 reads program data 1450 from HDD 1400 and executes it, but as another example, these programs may be obtained from other devices via external network 1550.
[0292] The present technology can also be configured as follows. (1) An acquisition unit that acquires an input query regarding video generation from a user; a scenario generation unit that generates scenario data related to video generation based on the input query; a code generation unit that generates a code for constructing 3D data based on the scenario data; a video data acquisition unit that acquires video data based on the code; An image generation system comprising: (2) an image quality improvement unit that executes an image quality improvement process to improve the image quality of the video data; The image generation system according to (1) further comprises: (3) The image quality improvement unit includes: The image quality of the video data is improved by executing the image quality improvement process based on a text prompt. (2) An image generation system. (4) The image quality improvement unit includes: The image quality improvement process is performed based on a text prompt that specifies a target of the image quality improvement process in the video data. An image generation system according to (2) or (3). (5) The image quality improvement unit includes: and performing the image quality improvement process based on a text prompt that specifies the object determined to require improvement. (4) An image generation system according to the present invention. (6) a display control unit that displays a storyboard based on the scenario data; The image generation system according to any one of (1) to (5), further comprising: (7) The storyboard is configured to display the video data for each cut of the video. (6) An image generation system according to the present invention. (8) a sound generating unit that generates sound data corresponding to the video data based on the scenario data and the video data; The image production system according to any one of (1) to (7), further comprising: (9) a text generating unit that generates text data indicating text to be displayed on a video displayed by the moving image data based on the scenario data; The image production system according to any one of (1) to (8), further comprising: (10) a logo generation unit that generates logo data indicating a logo to be displayed on a video image displayed based on the moving image data, based on the scenario data; The image production system according to any one of (1) to (9), further comprising: (11) The input query includes at least one of text, image, audio, and 3D data. The image generation system according to any one of (1) to (10). (12) a first output unit that outputs scenario generation information used by the scenario generation unit to generate the scenario data based on the input query; Further comprising: The scenario generation unit includes: The scenario data is generated based on the scenario generation information. The image generation system according to any one of (1) to (11). (13) The first output unit is generating a first prompt for generating the scenario data based on the input query as the scenario generation information; The scenario generation unit includes: generating the scenario data based on the first prompt The image generation system according to (12). (14) The first output unit is generating, based on the input query, first input information to be used as an input of a first model for generating the scenario data, as the scenario generation information; The scenario generation unit includes: The first input information generated using the input query is input to the first model, and the first model is caused to output the scenario data, thereby generating the scenario data. An image generation system according to (12) or (13). (15) a second output unit that outputs code generation information used by the code generation unit to generate code for configuring the 3D data based on the scenario data; Further comprising: The code generation unit The code is generated based on the code generation information. The image generation system according to any one of (1) to (14). (16) The second output section is generating, as the code generation information, a second prompt for outputting a code for configuring the 3D data based on the scenario data; The scenario generation unit includes: Generate the code based on the second prompt (15) An image generation system according to (15). (17) The second output section is generating second input information as the code generation information based on the input query, the second input information being used as an input of a second model for generating the code; The scenario generation unit includes: The second input information generated using the scenario data is input to the second model, and the second model is caused to output the code, thereby generating the code. An image generation system according to (15) or (16). (18) a reception unit that receives operations related to video editing from a user; Further comprising: The code generation unit The code is generated by editing based on the operation. The image generation system according to any one of (1) to (17). (19) The reception unit is Accepting the operation specifying a motion or camera movement using a sensor; The code generation unit Generate the code corresponding to the motion or camera movement indicated by the operation. The image generation system according to (18). (20) The reception unit is accepting the operation of selecting a plurality of cuts from the video data; The code generation unit Generate the code in which the parts corresponding to the plurality of cuts indicated by the operation have been changed An image generation system according to (18) or (19). (twenty one) Date information is associated with each cut of the video data, The code generation unit The content of the editing indicated by the operation is determined based on date information of each cut of the video data. The image generation system according to any one of (18) to (20). (twenty two) The reception unit is accepting the operation of designating an object to be changed from among the video data; The code generation unit Generate the code in which the 3D data of the object indicated by the operation has been changed. The image generation system according to any one of (18) to (21). (twenty three) the 3D data comprises a plurality of data sets; The code generation unit generating the code corresponding to at least one of the plurality of data sets by editing based on the manipulation; The image generation system according to any one of (18) to (22). (twenty four) The code generation unit Depending on the editing content indicated by the operation, execute either a process of updating a part of the plurality of data sets or a process of updating the plurality of data sets in its entirety. (23) An image generation system according to (23). (twenty five) an evaluation unit that generates information indicating an evaluation of at least one of the scenario data and the video data; The image production system according to any one of (1) to (24), further comprising: (26) The code generation unit generating said code based on said evaluation (25) An image generation system according to (25). (27) The scenario generation unit includes: The scenario data is generated based on the evaluation. An image generation system according to (25) or (26). (28) The code generation unit The code is generated based on the scenario data generated based on the evaluation. (27) An image generation system according to (27). (29) Obtaining an input query for video generation from a user; generating scenario data related to movie generation based on the input query; generating a code for constructing 3D data based on the scenario data; and acquiring video data based on said code; An image generation method comprising: (30) Obtaining an input query for video generation from a user; generating scenario data related to movie generation based on the input query; generating a code for constructing 3D data based on the scenario data; and acquiring video data based on said code; An image generation program that causes a computer to execute the above. [Explanation of symbols]
[0293] 1. Video Creation System 100 Image Generation Module 110 Input text analysis unit 120 Sensor Analysis Unit 130 Prompt Generation Unit 131 Scenario Generation Unit 132 Video Generation Unit 133 Sound Generation Unit 134 Text / Logo Generator 140 Image Generation Unit 141 USD generation section 142 Rendering Department 143 Video Refinement Department 150 Sound Generation Section 160 Text / Logo Generator 170 Composite Editorial Department 180 Evaluation Department 190 Client UI Module 200 Information Acquisition Module 210 Input text acquisition unit 220 Sensor Acquisition Unit 300 Sensor section 400 Client UI display section
Claims
1. An acquisition unit that acquires an input query regarding video generation from a user; a scenario generation unit that generates scenario data related to video generation based on the input query; a code generation unit that generates code, which is an editable computer program, for configuring 3D data and defining at least one of a scene and an object based on the scenario data; a video data acquisition unit that acquires video data based on the code; An image generation system comprising:
2. an image quality improvement unit that executes an image quality improvement process to improve the image quality of the video data; The video production system of claim 1 further comprising:
3. The image quality improvement unit includes: The image quality of the video data is improved by executing the image quality improvement process based on a text prompt. The video production system of claim 2 .
4. The image quality improvement unit includes: The image quality improvement process is performed based on a text prompt that specifies a target of the image quality improvement process in the video data. The video production system of claim 2 .
5. The image quality improvement unit includes: and performing the image quality improvement process based on a text prompt that specifies the object determined to require improvement. The video production system of claim 4.
6. a display control unit that displays a storyboard based on the scenario data; The video production system of claim 1 further comprising:
7. The storyboard is configured to display the video data for each cut of the video. The video production system of claim 6.
8. a sound generating unit that generates sound data corresponding to the video data based on the scenario data and the video data; The video production system of claim 1 further comprising:
9. a text generating unit that generates text data indicating text to be displayed on a video displayed by the moving image data based on the scenario data; The video production system of claim 1 further comprising:
10. a logo generating unit that generates logo data indicating a logo to be displayed on a video image displayed based on the moving image data, based on the scenario data; The video production system of claim 1 further comprising:
11. The input query includes at least one of text, image, audio, and 3D data. The video production system of claim 1 .
12. a first output unit that outputs scenario generation information used by the scenario generation unit to generate the scenario data based on the input query; Further comprising: The scenario generation unit includes: The scenario data is generated based on the scenario generation information. The video production system of claim 1 .
13. The first output section is generating a first prompt for generating the scenario data based on the input query as the scenario generation information; The scenario generation unit includes: generating the scenario data based on the first prompt 13. The video production system of claim 12.
14. The first output section is generating, based on the input query, first input information to be used as an input of a first model for generating the scenario data, as the scenario generation information; The scenario generation unit includes: The first input information generated using the input query is input to the first model, and the first model is caused to output the scenario data, thereby generating the scenario data.
13. The video production system of claim 12.
15. a second output unit that outputs code generation information used by the code generation unit to generate a code for configuring the 3D data based on the scenario data; Further comprising: The code generation unit The code is generated based on the code generation information. The video production system of claim 1 .
16. The second output section is generating, as the code generation information, a second prompt for outputting a code for configuring the 3D data based on the scenario data; The scenario generation unit includes: Generate the code based on the second prompt.
16. The image production system of claim 15.
17. The second output section is generating second input information as the code generation information based on the input query, the second input information being used as an input of a second model for generating the code; The scenario generation unit includes: The second input information generated using the scenario data is input to the second model, and the second model is caused to output the code, thereby generating the code.
16. The image production system of claim 15.
18. a reception unit that receives operations related to video editing from a user; Further comprising: The code generation unit The code is generated by editing based on the operation. The video production system of claim 1 .
19. The reception unit is Accepting the operation specifying a motion or camera movement using a sensor; The code generation unit Generate the code corresponding to the motion or camera movement indicated by the operation.
20. The video production system of claim 18.
20. The reception unit is accepting the operation of selecting a plurality of cuts from the video data; The code generation unit Generate the code in which the parts corresponding to the plurality of cuts indicated by the operation have been changed 20. The video production system of claim 18.
21. Date information is associated with each cut of the video data, The code generation unit The content of the editing indicated by the operation is determined based on date information of each cut of the video data.
20. The video production system of claim 18.
22. The reception unit is accepting the operation of designating an object to be changed from among the video data; The code generation unit Generate the code in which the 3D data of the object indicated by the operation has been changed.
20. The video production system of claim 18.
23. the 3D data comprises a plurality of data sets; The code generation unit generating the code corresponding to at least one of the plurality of data sets by editing based on the operation; 20. The video production system of claim 18.
24. The code generation unit Depending on the editing content indicated by the operation, execute either a process of updating a part of the plurality of data sets or a process of updating the plurality of data sets in its entirety.
24. The image production system of claim 23.
25. an evaluation unit that generates information indicating an evaluation of at least one of the scenario data and the video data; The video production system of claim 1 further comprising:
26. The code generation unit generating said code based on said evaluation 26. The image production system of claim 25.
27. The scenario generation unit includes: The scenario data is generated based on the evaluation.
26. The image production system of claim 25.
28. The code generation unit The code is generated based on the scenario data generated based on the evaluation.
28. The image production system of claim 27.
29. Obtaining an input query for video generation from a user; generating scenario data related to movie generation based on the input query; generating, based on the scenario data, code that is an editable computer program for constructing 3D data and defining at least one of a scene and an object; acquiring video data based on said code; An image generation method comprising:
30. Obtaining an input query for video generation from a user; generating scenario data related to movie generation based on the input query; generating, based on the scenario data, code that is an editable computer program for constructing 3D data and defining at least one of a scene and an object; acquiring video data based on said code; An image generation program that causes a computer to execute the above.
Citation Information
Patent Citations
Figure animation creation system
JP2003058906A
User-customized meta content providing system based on artificial neural network and method therefor
KR102508765B1
Automated story generation
US20110249953A1
Information processing device, information processing method, and program
WO2023002659A1
Cited By
Extreme rainstorm weather rendering method based on generative artificial intelligence
CN119206016A
Diffusion model virtual hairstyle changing method and system combined with Transform architecture
CN119251345A
Method and apparatus for generating digital human video based on large model, agent, electronic device, storage medium and computer program product
JP2025183260A
Programs, information processing devices, methods, and systems
JP2026144548A
Programs, information processing devices, methods, and systems
JP2026144948A
