Information processing methods, information processing systems, and programs

Abstract

This makes it possible to easily generate videos while minimizing the input burden on users. [Solution] The system accepts natural language text input, generates multiple sub-scenarios corresponding to multiple sections from the text, generates multiple section videos based on each sub-scenarios, concatenates the multiple section videos to generate a scenario video, and outputs the scenario video.

Description

Technical Field

[0001] The present invention relates to an information processing method, an information processing system, and a program.

Background Art

[0002] Patent Document 1 discloses an information processing apparatus capable of easily creating a picture manga.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that it is difficult to easily generate a video while suppressing the input burden of the user.

Means for Solving the Problems

[0005] According to one aspect of the present disclosure, an information processing method executed by an information processing system is provided. The information processing method receives an input of text data in natural language, generates a plurality of sub-scenarios corresponding to each of a plurality of sections from the text data, generates a plurality of section videos corresponding to each of the plurality of sub-scenarios based on each of the plurality of sub-scenarios, generates a scenario video by concatenating the plurality of section videos, and outputs the scenario video.

Brief Description of the Drawings

[0006] [Figure 1] FIG. 1 is a diagram showing an example of the system configuration of an information processing system. [Figure 2] FIG. 2 is a diagram showing an example of the hardware configuration of a server device. [Figure 3]Figure 3 shows an example of the hardware configuration of a client device. [Figure 4] Figure 4 shows an example of a functional configuration in an information processing system. [Figure 5] Figure 5 shows an example of the message creation (synopsis) screen. [Figure 6] Figure 6 shows an example of the message creation (section) screen. [Figure 7] Figure 7 is a flowchart showing an example of a process that generates sub-scenarios and enables section editing. [Figure 8] Figure 8 is a flowchart showing an example of the process for editing a section and changing its configuration. [Figure 9] Figure 9 is a flowchart showing an example of a process for generating spoken audio using voice selection. [Figure 10] Figure 10 is a flowchart showing an example of the process of generating and saving motion video. [Figure 11] Figure 11 is a flowchart showing an example of a process that generates section videos and outputs a completed video. [Figure 12] Figure 12 is a flowchart showing an example of a process that generates character images based on human images and makes them available for use in video generation. [Modes for carrying out the invention]

[0007] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below (including modified examples; the same applies hereinafter) can be combined with each other.

[0008] <Embodiment 1> 1. System Configuration Diagram Figure 1 shows an example of the system configuration of the information processing system 1000. As shown in Figure 1, the information processing system 1000 includes, as a system configuration, a server device 100, a client device 110, a model server device 120, a model server device 130, and a storage device 140. The server device 100 is an example of a computer. The server device 100, the client device 110, the model server device 120, the model server device 130, and the storage device 140 are connected to each other via a network 150. The network 150 may include one or both of the following: a WAN, a LAN, and / or the Internet. The network 150 is configured to allow devices connected to the network 150 to communicate with each other via wired and / or wireless connections. The information processing system 1000 may be a system that provides so-called SaaS functionality, or it may be a system that provides applications to the client device 110 and provides functionality in cooperation with the cloud-side server device 100.

[0009] The server device 100 performs the information processing described in this embodiment in response to requests from the client device 110. For example, the server device 100 receives natural language text data (synopsis, message, etc.), generates multiple sub-scenarios corresponding to multiple sections based on the text data, generates section videos based on each sub-scenarios, generates a scenario video (final video) by concatenating the multiple section videos, and outputs the generated scenario video. Details of the server device 100's processing will be explained later using flowcharts, sequence diagrams, etc.

[0010] In this specification, "scenario" refers to a data structure that describes the overall structure, content, or development of a video. This may include the full script written in natural language, a plot, or equivalent text data, as well as logical instruction data that defines the overall length of the video, the order of each component, and the transition conditions between each component. A "sub-scenario" refers to data that defines the content of expression within a given section, and is associated with each individual section that makes up a video. Specifically, it refers to a set of independently editable and manageable data that includes at least one of the following: the script for the audio to be read aloud in that section, the subtitle text to be displayed, the instructions (prompts) for the background image and video to be generated, the character's action instructions, and the estimated playback time of that section. A "section" is the smallest logical structural unit in video creation, dividing a video into chunks based on timeline or content. Each section is linked to its own sub-scenario, and the overall structure of the video is defined solely by its temporal sequence with other sections. Unlike the unstructured "clips" in traditional video editing, a section is defined as a unit linked to the division of a message (meaningful content). A "section card" is a user interface element that packages and displays multiple components related to a specific section (e.g., text input fields, still image thumbnails, motion settings, individual preview videos, etc.) as a single visual unit. Instead of each component existing individually, they are aggregated within a card-like framework, allowing users to complete the overall video structure through card-level operations (adding, deleting, rearranging, and editing content) without having to consider the timeline. "Motion information" refers to control parameters used to impart temporal changes to still scene images, or dynamic data generated based on those parameters. This includes camera work information that defines the operation of a virtual camera (pan, tilt, zoom, etc.), effect information that makes some pixels in the image appear fluid, and metadata for generating video that uses generation AI to impart specific actions (walking, hand gestures, etc.) to subjects in the image. In this specification, the term "section card" is not limited to a display form (card-type display), but may be a concept that includes a card data structure for generating such display. The card data structure may include a corresponding sub-scenario (or at least a part thereof) and section identification information in association, and a section card may be output based on this card data structure.

[0011] A sub-scenario may be stored as structured data containing information about the section in question. An example of a sub-scenario may include at least one of the following: (i) text for spoken audio (script), (ii) text for subtitle display, (iii) instruction information for background scene generation (prompt or template identifier including location, atmosphere, time of day, subject, art style, etc.), (iv) estimated playback time or target playback time for the section, (v) motion information (type or parameters such as pan, zoom, etc.), and (vi) whether or not a character is displayed or character identification information. This information is stored in the storage control unit 419 in association with section identification information (section number, order, etc.) and may be referenced as input in subsequent processes such as audio generation, background generation, motion generation, and section video synthesis.

[0012] Sub-scenarios can be represented as structured data (e.g., JSON, XML, or a table) that includes fields such as the following: (a) section_id (section identifier), (b) script_text (text for reading aloud), (c) subtitle_text (text for subtitles), (d) bg_prompt or bg_template_id (background generation instruction or template ID), (e) target_duration (target playback time), (f) motion_type and motion_param (motion type and parameters), (g) character_id (character specification), (h) transition_type (section transition specification). Some of these may be omitted, and when omitted, default values (for example, target_duration is a value calculated from the number of characters or phoneme length of script_text) may be adopted.

[0013] The structured data of the sub-scenario may include at least (a) section_id and (b) script_text as mandatory fields. section_id is an identifier that uniquely identifies a section and may be maintained unchanged throughout the processes of generating, editing, regenerating, and concatenating the section.

[0014] Also, as an example of the data type of each field, script_text and subtitle_text may be held as strings, target_duration as a numerical value in seconds, motion_type as an enumerated value (for example, pan, zoom, shake, none, etc.), and motion_param as a numerical parameter corresponding to the enumerated value (for example, speed, magnification, intensity, start timing, etc.).

[0015] Examples of default values adopted for omitted fields may be (i) target_duration is a value calculated by multiplying the number of characters or phoneme length of script_text by a coefficient, (ii) motion_type is none, and (iii) transition_type is a default transition (for example, cut).

[0016] Furthermore, target_duration may be subjected to clip processing (rounding to the upper limit value when exceeding the upper limit and rounding to the lower limit value when less than the lower limit) so as to fall within a predetermined range (for example, 1 second or more and 30 seconds or less).

[0017] The client device 110 is a device operated by a user (operator) of the information processing system 1000, and is a device that utilizes the functions provided by the server device 100. The client device 110 may be, for example, a notebook PC, a desktop PC, a tablet terminal, a smartphone, etc. The client device 110 can be any device as long as it can accept input operations from the operator (for example, video name, synopsis, number of section divisions, audio selection, BGM selection, section order change, merging instruction, etc.) and display the processing results from the server device 100 (for example, sub-scenarios, section cards, scene images, motion information, section video preview, completed video, etc.). Although only one client device 110 is shown in Figure 1 for the sake of simplicity, multiple client devices 110 may be included in the information processing system 1000.

[0018] The model server device 120 is a device that provides generative AI (e.g., a large-scale language model). The large-scale language model is a language model composed of an artificial neural network with many parameters and trained using a vast amount of text data. The model server device 120 performs processes such as generating the entire message (overall scenario) based on the input text data, dividing the entire message into sections to generate sub-scenarios, or directly generating sub-scenarios from the text data. Note that the functions of the model server device 120 may also be implemented in the server device 100.

[0019] The model server device 130 is a device that provides a media generation model service. The media generation model is a model that can generate new media (e.g., images, videos, audio, etc.) based on input data such as text and / or images. The model server device 130 performs at least some of the following: generating scene images according to sub-scenarios; generating character images (illustrations of people, etc.) based on photographs; generating audio for reading sub-scenarios based on selected audio; generating speech videos (so-called lip-sync videos) in which the mouth moves in sync with the speech; and generating background scene videos by adding motion (camera work, effects, AI-generated movement, etc.) to scene images. The functions of the model server device 130 may also be implemented in the server device 100.

[0020] The storage device 140 is a device for storing data and / or generation results used by the server device 100 in the generation process. The storage device 140 may be, for example, cloud storage (storage on a cloud infrastructure such as Azure). The storage device 140 may store, for example, section videos, scenario videos, thumbnails, BGM information selected by the user, generated character images, etc. Data that may contain personal information (for example, facial photographs, etc.) and temporary data in the process of generation may be configured to be deleted at a predetermined time.

[0021] Furthermore, the information processing system 1000 may include a server device that provides a service for a single multimodal model having the functions of both the model server device 120 and the model server device 130. In this case, the server device 100 may use the multimodal model service to perform tasks such as text generation, image generation, audio generation, and / or video generation.

[0022] Here, the information processing system described in the claims may consist of multiple devices or of one device. If the information processing system described in the claims consists of one device, an example of such a device is, for example, a server device 100. If the information processing system described in the claims consists of multiple devices, examples of the multiple devices are, for example, a distributed system that provides the functions of server device 100, or server device 100 and model server device 120, or server device 100 and model server device 130, or server device 100 and at least one client device 110.

[0023] 2. Hardware Configuration (1) Hardware configuration of server device 100 Figure 2 shows an example of the hardware configuration of server device 100. As shown in Figure 2, the server device 100 includes, as a hardware configuration, a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0024] The control unit 210 is one of the following: CPU (Central Processing Unit), MPU (Micro Processing Unit), SoC (System-on-a-Chip), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), GPU (Graphics Processing Unit), or at least two of these, and controls the entire server device 100 and performs various processes in response to natural language text data input from, for example, the client device 110. The control unit 210 performs processes such as generating multiple sub-scenarios corresponding to multiple sections based on text data, generating multiple section videos based on the sub-scenarios, generating a scenario video (completed video) by concatenating multiple section videos, controlling the output of the generated scenario videos, and generating user interface display data such as section cards.

[0025] The storage unit 220 is one of the following: HDD (Hard Disk Drive), ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drive), or a combination of at least two of these, and stores the program and the data used by the control unit 210 when it executes processing based on the program.

[0026] Examples of data used by the control unit 210 when executing processing include text data received from the client device 110 (video subject, synopsis, message text, etc.), scenarios or sub-scenarios generated from the text data (scripts for each section, etc.), information regarding the section structure (number of sections, section order, metadata corresponding to the start and end of each section, etc.), generation conditions (speaker type, reading speed, presence or absence of subtitles, aspect ratio, resolution, BGM type, sound effects, transition type, etc.), generation prompts or templates, models or parameters used for model inference (language generation model, image generation model, video generation model, speech synthesis model, lip-sync model, etc.), generated intermediate deliverables (scene images, audio data, subtitle data, motion information, section videos, etc.), completed video (scenario video), thumbnails, user identification information (account information, usage history, etc.), log data (generation requests, errors, processing time, etc.), etc.

[0027] The control unit 210 may manage each generated sub-scenario and the section video generated based on that sub-scenario in a storage unit 220 or the like, by associating them with section identification information (e.g., section number, display order, start / end metadata, etc.).

[0028] The storage unit 220 is an example of a non-temporary recording medium (storage medium) that is readable by a computer on which a program is recorded. In this specification, the data used by the control unit 210 when executing processing based on the program is described as being stored in the storage unit 220, but it may also be stored in the storage unit of another device (e.g., storage device 140, etc.) that can communicate with the server device 100. In other words, the data may be stored in the storage unit of any device as long as the control unit 210 can refer to and / or retrieve it.

[0029] The processing described herein is provided as a program for execution on a computer, which may be recorded on a non-temporary recording medium, distributed via a network 150, or provided as an update.

[0030] The control unit 210 executes processing based on the program stored in the memory unit 220, thereby realizing the functional configuration of the server device 100 shown in Figure 4 and the various flowcharts shown in Figures 7 to 12, which will be described later.

[0031] The communication unit 230 is a communication interface that connects the server device 100 to the network 150 and mediates communication with other devices. The communication unit 230 can be implemented using hardware that enables wired communication (e.g., Ethernet-compliant network interface card (NIC), LAN (Local Area Network) adapter, optical transceiver, etc.) and / or wireless communication (e.g., wireless communication modules such as Wi-Fi (Wireless Fidelity), Bluetooth (registered trademark), 5G (Fifth Generation), LTE (Long Term Evolution), NFC (Near Field Communication)), as well as circuits that control them.

[0032] Based on instructions from the control unit 210, the communication unit 230 performs data transmission and reception processing over the network 150 according to a predetermined communication protocol (e.g., TCP (Transmission Control Protocol) / IP (Internet Protocol), UDP (User Datagram Protocol), HTTP (Hypertext Text Transfer Protocol), WebSocket, MQTT (Message Queuing Telemetry Transport), etc.). For example, the communication unit 230 receives text data and generation conditions from the client device 110 and transmits the generated sub-scenarios, section video previews, and completed videos to the client device 110. The communication unit 230 may also transmit and receive inference requests and inference results (text, images, audio, video, etc.) between the model server device 120 and the model server device 130 (or the multimodal model providing device).

[0033] Furthermore, the hardware configurations of the control unit 210, storage unit 220, and communication unit 230 are not limited to one. For example, multiple control units may be included in the server device 100, or a configuration with multiple GPUs or a cluster configuration for distributed execution may be used for high-load processing such as video generation and encoding.

[0034] The hardware configurations of model server devices 120, 130, and 140 may be the same as those of server device 100 shown in Figure 2. However, each device may omit or add at least some of the components shown in Figure 2, depending on its intended use.

[0035] (2) Hardware configuration of client device 110 Figure 3 shows an example of the hardware configuration of the client device 110. As shown in Figure 3, the client device 110 includes, as a hardware configuration, a control unit 310, a storage unit 320, an input unit 330, an output unit 340, a communication unit 350, and an internal bus 360. The control unit 310, the storage unit 320, the input unit 330, the output unit 340, and the communication unit 350 are electrically connected via the internal bus 360.

[0036] The control unit 310 is a CPU, MPU, SoC, FPGA, ASIC, GPU, or a combination of at least two of these, and controls the entire client device 110 and performs various processes in response to, for example, operator input or data received from the server device 100. The control unit 310 performs display control of, for example, a text input screen, a section card screen, a sub-scenario display screen, a section video preview screen, and a completed video output screen, as well as processing the generation of text data and generation conditions (number of sections, order, BGM, speaker, subtitle settings, etc.) entered by the operator and sending them to the server device 100, and caching and playback control of received sub-scenarios and generation results (section videos, completed videos, etc.).

[0037] The storage unit 320 is one of the following: HDD, ROM, RAM, SSD, etc., or any combination thereof, and stores the program and data used by the control unit 310 when it executes processing based on the program. Examples of data stored include connection information to the server device 100, operator account information, entered text data (including data being entered), generation conditions (number of sections, section order, audio, BGM, subtitles, aspect ratio, etc.), generated results (at least in part) of received sub-scenarios, section videos and completed videos, temporary files for preview playback, and operation history and error logs.

[0038] The input unit 330 is a device that inputs information to the client device 110 in response to the operator's operation. The input unit 330 receives operation input made by the operator. The operation input is transferred to the control unit 310 via the internal bus 360 as a command signal. The control unit 310 can, if necessary, perform predetermined control and / or calculations based on the transferred command signal. The input unit 330 may be included in the housing of the client device 110 or it may be external. For example, the input unit 330 may be implemented as a touch panel display integrated with the output unit 340. When the input unit 330 is implemented as a touch panel display, the operator can input tap operations, swipe operations, etc. to the input unit 330. Instead of a touch panel display, the input unit 330 can be a switch button, mouse, trackpad, keyboard, etc. The operator may, for example, input text data, specify the number of sections, rearrange the section order, modify sub-scenarios, issue a start command for generation, issue a preview playback command, and issue a command for outputting the completed video (save, share, etc.) via the input unit 330.

[0039] The output unit 340 is a display unit, such as a display, that outputs (displays) information as a GUI (Graphical User Interface) screen that can be operated by the operator. The output unit 340 may be included in the housing of the client device 110 or it may be external. More specifically, the output unit 340 may be implemented as a display device such as a liquid crystal display, an organic EL display, or a plasma display. The output unit 340 may display, for example, input text data, generated sub-scenarios, section cards, previews of section videos, and completed videos.

[0040] The communication unit 350 is a communication interface that connects the client device 110 to the network 150 and mediates communication with other devices. The communication unit 350 can be implemented, for example, by hardware that realizes wired communication (e.g., Ethernet-compliant NIC, LAN adapter, optical transceiver, etc.) and / or wireless communication (e.g., wireless communication modules such as Wi-Fi, Bluetooth®, 5G, LTE, NFC, etc.), as well as circuits that control them.

[0041] Based on instructions from the control unit 310, the communication unit 350 performs data transmission and reception processing over the network 150 according to a predetermined communication protocol (e.g., TCP / IP, HTTP, WebSocket, MQTT, etc.). For example, the communication unit 350 transmits text data and generation conditions entered by the operator to the server device 100 and receives sub-scenarios, section video previews, and completed videos from the server device 100.

[0042] 3. Functional Configuration (1) Functional configuration of the information processing system 1000 Figure 4 is a block diagram showing an example of the functional configuration in the information processing system 1000. As shown in Figure 4, the information processing system 1000 includes, for example, an input receiving unit 401, a generation condition receiving unit 402, a scenario generation unit 403, a section division unit 404, a sub-scenario generation unit 405, a section number control unit 406, a UI data generation unit 407, a sub-scenario editing receiving unit 408, a voice selection receiving unit 409, a voice data management unit 410, a voice synthesis unit 411, a character generation unit 412, a lip-sync generation unit 413, a background scene generation unit 414, a section video generation unit 415, a linking unit 416, an output control unit 417, a model linking unit 418, a storage control unit 419, etc., which are implemented by the server device 100. At least some of these functions may be implemented by the client device 110, or by utilizing the services of the model server device 120 and the model server device 130.

[0043] The input receiving unit 401 accepts text data in natural language. The text data may include messages to be included in the completed video, summaries, lists of events, or descriptions of each scene.

[0044] The generation condition receiving unit 402 accepts various conditions to be used in the generation process. These conditions may include the number of sections, section order, specification of audio (speaker), merging instructions, presence or absence of subtitles, aspect ratio, resolution, BGM type, sound effects, transition type, etc.

[0045] The scenario generation unit 403 generates a scenario from text data using a generation AI (e.g., a large-scale language model). The scenario is a text or sequence of instructions that corresponds to the overall structure of the completed video and may contain content that can be divided into multiple sections.

[0046] The section division unit 404 divides the scenario generated by the scenario generation unit 403 into sections, thereby generating multiple sub-scenarios corresponding to each of the multiple sections. The unit of section division may be a division such as introduction, event 1, event 2, conclusion, etc.

[0047] The sub-scenario generation unit 405 generates multiple sub-scenarios corresponding to multiple sections from the text data. The sub-scenario generation unit 405 may (a) divide the scenario obtained by the scenario generation unit 403 using the section division unit 404 to generate sub-scenarios, or (b) directly generate sub-scenarios from the text data using a generation AI.

[0048] If the sub-scenario generation unit 405 adopts the configuration of (a), it may send a request to the model server device 120 via the model linkage unit 418 to generate a full message (overall scenario) from text data, and the received overall scenario may be divided into sections by the section division unit 404 to generate multiple sub-scenarios corresponding to each of the multiple sections.

[0049] In this case, the sub-scenario generation request in the process shown in Figure 7 (for example, S705) may be configured to include an overall scenario generation request and a section division request for the overall scenario.

[0050] The section count control unit 406 controls the generation conditions of the subscenario generation unit 405 so that a number of subscenarios corresponding to the number of sections received by the generation condition receiving unit 402 is generated. For example, if the number of sections is "4", it may be controlled to generate 4 subscenarios.

[0051] The UI data generation unit 407 generates screen data for outputting a screen containing multiple section cards, each corresponding to a multiple section. Each of the multiple section cards is configured to display a sub-scenario corresponding to that section card. Here, "output" includes transmitting data for display on the display unit of the client device 110.

[0052] The section division unit 404 may divide the text data into multiple sections according to the following rules, for example, without using a generation AI. (1) For the number of sections N specified by the operator, the text data is divided into sentence units or period units, and importance levels (e.g., TF-IDF, presence or absence of named entities, presence or absence of conjunctions) are assigned to each unit. (2) Based on the total number of characters or estimated speech length (number of characters × coefficient), units are allocated using greedy or dynamic programming so that the target quantity for each section is equal. (3) The division boundaries should be set prioritizing paragraph breaks, heading symbols, or chronological words (such as "next" or "after that"). This allows for section-based splitting, independent of the inference results of the generating AI.

[0053] As a typical example of the allocation process in (2), the section division unit 404 may perform a greedy allocation by scanning the sentence (or period) column from the beginning, adding units as long as the cumulative value of the estimated speech length allocated to the current section does not exceed the target amount, and if it exceeds the target amount, dividing at the immediately preceding boundary candidate (paragraph break, heading symbol, or immediately before a time-series word, etc.) and moving to the next section. After all sections have been allocated, if the deviation of the estimated speech length of each section exceeds a predetermined threshold, adjustments may be made by moving units between adjacent sections.

[0054] The sub-scenario generation unit 405 may include (i) a mode in which it generates script_text etc. for each section using a generation AI, as well as (ii) a mode in which it generates using templates and rules. For example, in the template method, a standard text box is maintained for each section type (introduction, events, conclusion, etc.), and key phrases (noun phrases, dates, locations, etc.) extracted from the segmented text are embedded in the slots to generate script_text and subtitle_text. Furthermore, if the inference result of the generating AI does not satisfy predetermined conditions (for example, target_duration range, prohibited words, character limit, JSON schema consistency), the sub-scenario generation unit 405 may (a) regenerate (prompt or adjust conditions) or (b) fall back to the template method.

[0055] The UI data generation unit 407 may output to the output unit 340 of the client device 110 data for each section card, including data for displaying at least a portion of the sub-scenario associated with that section card (e.g., text to be read aloud, text for subtitles, etc.).

[0056] The screen data generated by the UI data generation unit 407 includes a collection of card data corresponding to multiple section cards, and each card data may be configured to include a sub-scenario (or at least a part thereof) associated with that section card as data. For example, the card data may hold at least one of the sub-scenario's fields (text for reading aloud, text for subtitles, background generation instructions, target playback time, motion information, etc.) within the card data, and the client device 110 displays the section card based on the card data.

[0057] The sub-scenario editing reception unit 408 is configured to allow editing of sub-scenarios corresponding to section cards. For example, it may receive the edited sub-scenario in response to editing operations via the screen of the client device 110 and reflect it in the sub-scenario generation unit 405 or the storage control unit 419.

[0058] The voice selection reception unit 409 accepts voice selection operations via the screen. The voices to be selected may include multiple pre-prepared speaker voices or voices uploaded by the user.

[0059] The audio data management unit 410 registers the audio uploaded by the user and makes it available for selection on the screen. For example, the audio data management unit 410 may issue an identifier corresponding to the uploaded audio, and the audio selection receiving unit 409 may be configured to present the identifier as an option for selection.

[0060] The speech synthesis unit 411 generates a spoken audio by reading a sub-scenario through speech synthesis. The speech synthesis unit 411 may also generate the spoken audio based on the audio selected by the audio selection reception unit 409. In this case, the speech synthesis unit 411 may use an uploaded audio managed by the audio data management unit 410 as the speaker.

[0061] The background scene generation unit 414 generates background scene videos to be included in the section video. Depending on the corresponding sub-scenario, the background scene generation unit 414 may determine the content of the background scene video (location, atmosphere, time of day, etc.) and generate the background scene video using a generation AI.

[0062] The background scene generation unit 414 is not limited to generating moving images using a generation AI, but may also generate a background scene moving image by (i) preparing a background still image, (ii) applying virtual camera transformations such as panning and zooming as a time function, and (iii) superimposing effects such as particles as necessary. In this case, motion_type and motion_param can be expressed as, for example, pan amount (px / sec), zoom magnification, oscillation frequency, etc.

[0063] The character generation unit 412 generates character images included in the section video. The character generation unit 412 may generate character images based on a photograph. The photograph may be a photograph of a person uploaded by the user. In data management during character generation, this system implements exclusive data retention control to protect privacy. Specifically, it stores user-uploaded images of people (photo data) in temporary memory, executes an API for illustration processing, and immediately after confirming that the generated character image (illustration data) has been successfully output, physically deletes the original person image data. At this time, the GUI is also controlled so that the original person image and the generated character image are not displayed simultaneously on the same screen. By setting the completion of the deletion of person image data as a trigger for execution of transition operations after the completion of character image generation or adoption, a state is forcibly created in both cloud and local environments where personally identifiable real photographic data and abstracted illustration data do not exist simultaneously.

[0064] The deletion of person image data can be implemented, for example, by storing the person image data in a temporary area separate from persistent storage, and after the completion of character image (illustration data) generation is confirmed, invalidating the reference to the temporary area and deleting it from the temporary area. Furthermore, screen transitions on the client device 110 may be controlled to be permitted only after receiving a deletion completion response (or deletion job completion notification) from the server device 100. This makes it possible to minimize the retention period of person image data.

[0065] The lip-sync generation unit 413 generates mouth movements based on the character image and spoken audio so that the character image's mouth moves in response to the reading of a sub-scenario corresponding to the section video. The mouth movements can be realized as changes in mouth shape synchronized with the time changes of the spoken audio.

[0066] The section video generation unit 415 generates multiple section videos corresponding to each of the multiple sub-scenarios, based on each of the multiple sub-scenarios. The section video generation unit 415 may also generate section videos that include spoken audio. Furthermore, the section video generation unit 415 may also generate section videos that include character images and mouth movements. In addition, the section video generation unit 415 may also generate section videos that include background scene animations.

[0067] The section video generation unit 415 may generate a section video including the spoken audio by synthesizing the spoken audio generated by the speech synthesis unit 411 with video elements (e.g., background scene animation, character images, etc.) based on the corresponding sub-scenario in a time-synchronized manner.

[0068] The section video generation unit 415 may generate a speech video in which the mouth of the character image moves based on the mouth movement generated by the lip-sync generation unit 413, and superimpose the speech video onto the background scene video to output it as a section video including the character image.

[0069] The section video generation unit 415 may generate a section video that includes a background scene video corresponding to the corresponding sub-scenario by using the background scene video generated by the background scene generation unit 414 as a video element of the section video.

[0070] The section video generation unit 415 obtains at least script text, instruction information for background scene generation, and (if present) motion information from the corresponding sub-scenario, and generates or obtains at least one of the following: spoken audio based on the script text, background scene video, and motion video. Then, using the spoken audio as the time axis reference, it may synthesize it with the background scene video (or motion video) in time synchronization and output it as a section video in a predetermined video format. If subtitles are specified, the subtitle text may be generated in association with the display timing information, and these subtitles may be superimposed onto the video frames of the section video.

[0071] The section video generation unit 415 may use the duration of the spoken audio as the reference for the time axis and arrange and synthesize each video element such as background scene video, character images, mouth movements, and subtitles so as to match the said duration.

[0072] As an example of fallbacks for missing material, the following may be implemented to allow the section video generation process to continue even if some of the material is missing: (a) if background scene video has not been generated, a background still image or a monochrome background will be used and held for a predetermined time; (b) if motion information is not specified, a still display (or a default slight zoom) will be applied; (c) if mouth movement has not been generated, a still character image will be superimposed; (d) if subtitle display is disabled, the superimposition of subtitles will be omitted.

[0073] As an example of output specifications, the video generation section 415 may generate a video file with a predetermined resolution, frame rate, and codec (e.g., MP4, 30fps). These are examples and may be changed depending on the generation conditions.

[0074] The linking unit 416 generates a scenario video by linking multiple section videos. The linking unit 416 may also generate a scenario video by linking multiple section videos based on a linking instruction received by the generation condition receiving unit 402. The linking instruction may include instructions for determining the order of section videos, applying transitions, or starting the linking process. The linking unit 416 recalculates the metadata (start time and end time) associated with each section video when the order of the section cards is changed, and recombines the video files in the background. At this time, since each section is maintained as an independent video file, even if only a specific card is edited and regenerated, there is no need to re-render the entire video from scratch, thus achieving both reduced processing load and immediate preview. The linking unit 416 may store sequence information (hereinafter referred to as "linking sequence information") that defines the linking order of multiple section videos as a sequence value associated with the display order of the section cards or section identification information. Link order information is managed in association with the display order of section cards, and if the display order of section cards is changed, the link order information is updated according to the changed display order. If the generation condition receiving unit 402 receives the result of a change in the section order, the linking unit 416 may update the linking order information according to the change result and generate a scenario video by linking multiple section videos according to the updated linking order information.

[0075] With respect to the spoken audio generated by the speech synthesis unit 411, the section video generation unit 415 may obtain the length (duration) of the audio, and if the difference with target_duration exceeds a threshold, it may adjust the length of the section video by (a) adjusting the reading speed parameter, (b) inserting a silent section, or (c) adjusting the subtitle display time. Alternatively, when adding subtitles, the subtitle text may be divided into sentences or periods, and the display time for each unit may be proportionally allocated to the audio length to generate a subtitle timestamp sequence (start time, end time).

[0076] The combination instructions received by the generation condition receiving unit 402 may include, for example, at least one of the following: (i) confirmation of the order of section cards, (ii) start of linking, (iii) confirmation of the application of transitions or background music, and (iv) output instruction for the completed video. The linking unit 416 may recalculate timestamps (start time and end time) based on the section order in response to the combination instructions and link multiple section videos in that order to generate a scenario video.

[0077] In this specification, "combine instruction" refers to operation information or state information that acts as a trigger to start or confirm the process of linking multiple section videos to generate a scenario video (linking process). A combine instruction can be implemented, for example, by pressing the combine button 660, selecting "Combine / Output" in the menu, or by notifying that a predetermined condition (for example, completion of generation of all section videos, or confirmation of the order by the operator) has been met. The linking unit 416 performs the linking of multiple section videos or updates the provisional linking result when a linking instruction is received. Therefore, if a linking instruction is not received, the linking unit 416 may not perform the linking process even if the section videos have already been generated (however, this does not exclude the configuration in which a temporary linking result for preview is internally generated).

[0078] The output control unit 417 outputs the generated scenario video. The output may include transmission to the client device 110, storage in the storage device 140, or making it available for download in a predetermined format.

[0079] The model linkage unit 418 sends an inference request to the model server device 120 and / or the model server device 130 and receives the inference result. The inference result may include a scenario, sub-scenario, background scene video, character images, spoken audio, or section video. The inference request transmitted by the model linkage unit 418 may include at least information identifying the target section and a sub-scenario or part thereof (script text, background generation instructions, motion information, etc.). The inference request may further include at least one of the following parameters: resolution, frame rate, video length, aspect ratio, speaker type, reading speed, and generation style. The model linkage unit 418 may receive images, videos, audio, mouth movement data, or videos synthesized therefrom as inference results.

[0080] The storage control unit 419 stores data used in the generation process and the generation results, and reads them as needed. The stored data may include text data, sub-scenarios, generation conditions, spoken audio, background scene videos, character images, section videos, and scenario videos.

[0081] 4. Information Processing (1) Overview of the process The input receiving unit 401 accepts text data in natural language. The generation condition receiving unit 402 may accept generation conditions such as the number of sections, audio, and linking instructions. The sub-scenario generation unit 405 generates multiple sub-scenarios corresponding to each of the multiple sections based on the input text data. The section video generation unit 415 generates multiple section videos corresponding to each of the multiple sub-scenarios based on each of the multiple sub-scenarios. The linking unit 416 generates a scenario video by linking the multiple section videos. The output control unit 417 outputs the generated scenario video.

[0082] This process allows users to generate videos organized into sections simply by inputting natural language text data, and then easily obtain a complete video by linking multiple section videos together. As a result, it becomes easier to create a complete video that is relevant to the content without having to perform cumbersome tasks such as timeline editing.

[0083] (2) Details of the process The flowcharts in Figures 7 to 12 below illustrate typical sequences and processing branches for the sake of explanation. In actual execution, each step may be selectively executed in response to an operation event on the screen of the client device 110 (e.g., button press, selection confirmation, save instruction, generation start instruction, merging start instruction, etc.) or the achievement of a predetermined state (e.g., generation complete, all sections completed, acquisition of predetermined materials completed, etc.). Therefore, not all steps shown in each figure are always executed, and if a predetermined function is not selected, or if already generated data is reused, some of the steps shown in each figure may be omitted or replaced with equivalent processing. Furthermore, each step is not limited to the order shown in the diagram, and may be performed sequentially or in parallel. In addition, at least a portion of each step is not limited to being performed immediately in synchronization with user operation, but may be performed as background or asynchronous processing, and configured so that progress, completion notifications, or reference information of the generated product are later reflected on the screen.

[0084] Figure 5 shows an example of the message creation (synopsis) screen 500. The UI data generation unit 407 transmits screen data containing the following elements to the client device 110. Upon receiving the screen data, the client device 110 generates a screen based on the screen data and displays it on the output unit 340. The screen data includes the following elements: a display area 510, an input field 520, a selection area 530, an input field 540, a character counter 550, a selection area 560, an execute button 570, and a background music list 580. Display area 510 is the thumbnail display area for the character illustration associated with the video. Input field 520 is for entering the video title. Selection area 530 is the selection area for the voice to be used for speaking (pressing it will play or stop the sample voice). Input field 540 is for entering the synopsis. The character count of 550 is the character count of the synopsis. Selection area 560 is the selection area for the number of section divisions. Execute button 570 is the button to execute section splitting. BGM List 580 is a list of background music tracks. The BGM can be played or stopped by pressing a button. The voice selection area for speech may display pre-prepared speaker voices as well as uploaded voices registered by the voice data management unit 410 as selection candidates, and accept selection operations. The UI data generation unit 407 may generate screen data for transitioning to a screen for registering a person image (photo) and generating a character image, in response to an operation on the thumbnail display area of ​​the person illustration.

[0085] Figure 6 shows an example of the message creation (section) screen 600. The UI data generation unit 407 transmits screen data containing the following elements to the client device 110. Upon receiving the screen data, the client device 110 generates a screen based on the screen data and displays it on the output unit 340. The screen data includes, as elements, a display area 610, display buttons 620, multiple sections 630, editing areas 640 for each section, an add button 650, and a composite button 660. Display area 610 is the display area for the selected audio. The display button 620 is used to display the synopsis entered in input field 540 on the message creation (synopsis) screen 500. The multiple sections 630 are each separate sections that are divided by the selection of the execute button 570 on the message creation (synopsis) screen 500. The editing area 640 for each section is the editing area (editing window) for editing the message for that section. Button 650 is a button that adds an empty section. The composite button 660 is used to combine videos generated in each of the multiple sections in the order of the sections arranged from top to bottom on the message creation (section) screen 600. Each of the multiple sections 630 is configured to allow its order to be changed by dragging it up and down. The UI data generation unit 407 may activate the composite button 660 after the generation of each section video is complete.

[0086] Figure 7 is a flowchart illustrating an example of a process for generating sub-scenarios and enabling section editing. Each process in Figure 7 can be selectively executed in response to an operation event on the message creation (synopsis) screen 500 or the establishment of a predetermined state. For example, if the input of the synopsis or the setting of the generation conditions is not yet determined, the system may transition to another screen after the execution of (S702) and / or (S703), and if there is no instruction to execute section division (e.g., pressing the execute button 570), (S704) to (S707) will not be executed. Furthermore, in an embodiment where an already saved sub-scenarios are reused, the model inference in (S705) and (S706) may be omitted.

[0087] (S701) The UI data generation unit 407 transmits the screen data shown in Figure 5 to the client device 110.

[0088] (S702) The input receiving unit 401 may receive text data (summary) from the client device 110 and send it to the storage control unit 419 for storage.

[0089] (S703) The generation condition receiving unit 402 may receive the video name, audio selection, number of section divisions, BGM selection, etc. from the client device 110 and send it to the storage control unit 419 for storage.

[0090] (S704) The section number control unit 406 sets the generation conditions for the sub-scenario generation unit 405 according to the number of section divisions.

[0091] (S705) The sub-scenario generation unit 405 sends a sub-scenario generation request to the model linkage unit 418 based on the text data and generation conditions. The model linkage unit 418 generates inference request data for the model server device 120 based on the sub-scenario generation request received from the sub-scenario generation unit 405. The inference request data may include at least the number of sections (or identification information for each section), the output format (e.g., a format including the structured fields of the sub-scenario), and instructions for sub-scenario generation (prompt or template identifier). The model linkage unit 418 sends the generated inference request data to the model server device 120.

[0092] (S706) The model linkage unit 418 receives multiple sub-scenarios from the model server device 120. The model linkage unit 418 analyzes the received sub-scenarios and formats them as sub-scenario data associated with section identification information. For example, the model linkage unit 418 associates a number of sub-scenarios corresponding to the number of sections with each section identification information. If the received result does not satisfy predetermined format conditions (e.g., match with the number of sections, no missing required fields, upper limit on the number of characters, etc.), the model linkage unit 418 may adjust the generation conditions or inference request data and make a re-request, or fall back to the template method.

[0093] The prescribed formal conditions may include at least one of the following: (a) The received data must be parsable in a predetermined structured format (e.g., JSON). (b) The number of sub-scenarios received matches the number of sections received. (c) Each sub-scenario must include the required fields (at least section_id and script_text). (d) The number of characters in script_text or target_duration must fall within the specified range.

[0094] If the formal conditions are not met, the model linkage unit 418 may resend the inference request data for re-request to the model server device 120, including the inconsistent content (e.g., missing field names, expected number, number of characters exceeding the upper limit, etc.). For example, if a required field is missing, the regeneration request may be made with a clear indication that the field must be included. Furthermore, if the number of re-requests reaches a threshold, the system may fall back to the template method.

[0095] (S707) The sub-scenario generation unit 405 sends multiple sub-scenarios to the storage control unit 419 for storage.

[0096] (S708) The UI data generation unit 407 generates the screen data shown in Figure 6 and sends it to the client device 110.

[0097] The model linkage unit 418 performs the following operations on the response (subscenario column) received from the model server device 120: (i) parsing the structured syntax, (ii) verifying the existence of required fields (at least section_id and script_text), (iii) verifying that the number of received items matches the number of sections, and (iv) associating each section_id with the section identification information received from the client device 110. The verified subscenarios are then normalized into a predetermined schema and stored in the storage control unit 419. Furthermore, the UI data generation unit 407 generates card data that includes sub-scenario elements to be displayed on each section card (e.g., text for reading aloud, text for subtitles, target playback time) based on the saved results, and transmits the card data to the client device 110.

[0098] Figure 8 is a flowchart illustrating an example of section editing and configuration change processing using Figure 6. Each process in Figure 8 may be selectively executed in response to an operation event on the message creation (section) screen 600. Specifically, (S801) and (S802) may be omitted if no editing operation is performed. Also, (S803) may be omitted if no section order is changed. Furthermore, (S804) and (S805) may be omitted if no empty section addition operation is performed. Note that screen updates by (S806) may be executed when changes occur in the section card group due to each operation. In addition, saving editing results, saving order change results, and screen updates may be executed sequentially or in batches for the same operation, and consistency processing (e.g., updating link order information or recalculating timestamps) may be performed in the background. Each process in Figure 8 may be selectively executed in response to an operation event on the message creation (section) screen 600 or the establishment of a predetermined state.

[0099] (S801) The sub-scenario editing reception unit 408 receives editing operation information from the client device 110.

[0100] (S802) The sub-scenario editing reception unit 408 generates the edited sub-scenario and sends it to the storage control unit 419 for saving. The sub-scenario editing reception unit 408 may normalize the edited sub-scenario to a predetermined field structure (e.g., script_text, subtitle_text, etc.) and supplement any missing required fields with default values. Furthermore, it may prevent mismatching to other sections by saving the edited section while retaining its section identification information.

[0101] (S803) The generation condition receiving unit 402 may receive the section order change result from the client device 110 and send it to the storage control unit 419 for storage. The generation condition receiving unit 402 or the linking unit 416 may update the linking order information according to the order change result and recalculate the start time and end time associated with each section video.

[0102] (S804) The generation condition receiving unit 402 may receive an empty section addition operation from the client device 110.

[0103] (S805) The sub-scenario generation unit 405 may generate a sub-scenario corresponding to an empty section, or it may send it to the storage control unit 419 for storage. The sub-scenario generation unit 405 may assign new section identification information to an empty section and generate an empty or template-based sub-scenario that includes at least one of the default fields (e.g., script_text, subtitle_text, or target_duration).

[0104] (S806) The UI data generation unit 407 may also send the updated screen data of the section card group to the client device 110.

[0105] Figure 9 is a flowchart illustrating an example of the process of generating spoken audio using the voice selection shown in Figures 5 and 6. For example, if a pre-prepared speaker voice is selected, steps (S902) and (S903) related to the uploaded voice may be omitted. Also, if the voice selection information is already stored as a generation condition, step (S904) may be omitted. Furthermore, in the case of reusing already generated spoken audio, the regeneration of steps (S905) and (S906) may be omitted, and reference information for the saved voice may be used. Each process in Figure 9 may be selectively executed in response to an operation event related to voice selection or uploading, or to the establishment of a predetermined state.

[0106] (S901) The voice selection reception unit 409 receives voice selection information from the client device 110.

[0107] (S902) The voice data management unit 410 may receive uploaded voice data from the client device 110.

[0108] (S903) The audio data management unit 410 assigns an identifier to the uploaded audio data and sends it to the storage control unit 419 for storage.

[0109] (S904) The generation condition receiving unit 402 may hold the voice selection information as a generation condition and send it to the storage control unit 419 for storage.

[0110] (S905) The speech synthesis unit 411 sends a request to the model linkage unit 418 to generate a read-aloud voice based on the sub-scenario and voice selection information. The model linkage unit 418 may generate inference request data for the model server device 130 based on the speech output request. The inference request data may include at least information identifying the target section, text for speech output, speaker identification information, and output speech format (e.g., sampling frequency, bit depth, number of channels, or codec). The inference request data may also include at least one of the parameters such as speech speed, emotion style, and silence insertion tolerance. The model linkage unit 418 may transmit the generated inference request data to the model server device 130.

[0111] (S906) The model linkage unit 418 receives the read-aloud audio from the model server device 130. The model linkage unit 418 may analyze the received read-aloud audio, format it into a predetermined audio format (e.g., sampling frequency, bit depth, number of channels, codec), and pass it to the speech synthesis unit 411 in association with information identifying the target section. If the reception result does not meet predetermined conditions (e.g., audio length, silent interval, noise level, etc.), the generation conditions or inference request data may be adjusted and a re-request may be made.

[0112] (S907) The speech synthesis unit 411 transmits the spoken audio to the storage control unit 419 for storage. When the speech synthesis unit 411 detects that it has finished saving the spoken audio associated with the information identifying the target section, it notifies the UI data generation unit 407 of the status update for that section (for example, "speech generation complete"). The UI data generation unit 407 then sends the status update information and audio data reference information (file path or URL, etc.) to the client device 110 to enable preview playback.

[0113] The result of selecting an effect (specifying the application of motion) may be accepted as operation information that includes information identifying the scene image or section to which the effect will be applied, the type of motion (e.g., pan, tilt, zoom, shake, particles, etc.), and at least one of the parameters such as the intensity, duration, and start timing of the motion.

[0114] Figure 10 is a flowchart illustrating an example of the process of generating and saving a motion video. Each step in Figure 10 is an example of what is executed when the motion application (effect application) function is selected and a save operation is performed. If motion application is not specified, or if a default motion (or still display) is used, at least some of steps (S1002) to (S1005) may be omitted. Furthermore, in the case of reusing an already saved motion video, the generation processes in (S1003) and (S1004) may be omitted.

[0115] (S1001) The UI data generation unit 407 transmits screen data related to motion video generation to the client device 110. At this time, the screen data related to motion video generation transmitted by the UI data generation unit 407 may include, for example, at least one of the following: an overview display area for the target section, a display area for the section list or target selection, a generation mode switching area, a display area for the selected scene image, an effect selection area, a back operation area, and a save operation area. The generation condition receiving unit 402 may receive the effect selection result in association with information identifying the section or scene image to which it will be applied. The section video generation unit 415 may start generating a motion video based on the selected effect in response to a save operation.

[0116] (S1002) The generation condition receiving unit 402 may receive the effect selection result from the client device 110.

[0117] (S1003) The section video generation unit 415 may send a motion video generation request to the model linkage unit 418 based on the effect selection result and the scene image. The model linkage unit 418 may generate inference request data for the model server device 130 based on the motion video generation request. The inference request data may include at least information identifying the section or scene image to be applied, an input scene image (or reference information for the scene image), a motion type, and motion parameters (e.g., intensity, duration, start timing, etc.). The inference request data may further include at least one of the output video formats (e.g., resolution, frame rate, video length, or codec). The model linkage unit 418 may transmit the generated inference request data to the model server device 130.

[0118] (S1004) The model linkage unit 418 receives motion video from the model server device 130. The model linkage unit 418 may analyze the received motion video, format it into a predetermined video format (e.g., resolution, frame rate, video length, codec), and pass it to the section video generation unit 415, associating it with information identifying the section or scene image to which it applies. If the reception result does not meet predetermined conditions, the generation conditions or inference request data may be adjusted and a re-request may be made.

[0119] (S1005) The section video generation unit 415 sends the motion video to the storage control unit 419 for storage.

[0120] (S1006) The UI data generation unit 407 may also present a display status to the client device 110 indicating that saving is complete or that previewing is possible.

[0121] The instruction to start section video generation, the instruction to preview playback, and the operation to start combining may be received as operation information corresponding to any user operation such as pressing a button, selecting a menu, or entering a shortcut, and such operation information may be stored in the storage control unit 419 as generation conditions.

[0122] Figure 11 is a flowchart showing an example of a process for generating section videos and outputting a completed video. Each process in Figure 11 can be selectively executed in response to an operation event such as a section video generation start instruction or a merging start instruction, or in response to the establishment of a predetermined state. For example, if the generation start operation is not performed, (S1102) to (S1108) will not be executed. Also, in a configuration that does not use character images, (S1104) may be omitted, and in a configuration that does not use lip-sync, (S1105) may be omitted. Furthermore, if materials such as background scene videos or motion videos have already been generated, at least a part of the corresponding generation process (for example, (S1103)) may be omitted, and reference information of the saved materials may be obtained and used. In addition, if the merging start operation is not performed, (S1111) and (S1112) will not be executed, and the completed video will not be output. Note that the activation by (S1110) and the merging process by (S1112) may be executed in the background, and the provisional merging result may be updated sequentially.

[0123] (S1101) The UI data generation unit 407 transmits screen data related to section video generation to the client device 110. At this time, the screen data related to section video generation transmitted by the UI data generation unit 407 may indicate at least one of the following states: the state before generation starts (displaying the generation start operation area), the state during generation (displaying a display indicating the progress), and the state after generation is complete (transitioning to a previewable display). The generation condition receiving unit 402 may receive operation information corresponding to the generation start operation. The section video generation unit 415 may generate a section video by synthesizing a speech video and a motion video according to the operation information. Furthermore, the client device 110 may be configured to display at least one of a preview of the generated section video in a form without superimposed character images and a preview in a form with superimposed character images.

[0124] (S1102) The section video generation unit 415 acquires sub-scenarios, spoken audio, motion videos, and generation conditions from the storage control unit 419.

[0125] (S1103) The background scene generation unit 414 may generate a scene image according to a sub-scenario and send it to the storage control unit 419 for storage. The background scene generation unit 414 may send a scene image generation request to the model linkage unit 418 based on the sub-scenario (e.g., bg_prompt or bg_template_id) and generation conditions. The model linkage unit 418 may generate and send inference request data (e.g., resolution, aspect ratio, style, output format, etc.) to the model server device 130 based on the scene image generation request. The model linkage unit 418 may receive the scene image from the model server device 130, analyze the received scene image, format it into a predetermined format (e.g., image format, resolution, color mode), associate it with information identifying the target section, and pass it to the background scene generation unit 414. If the received result does not meet the predetermined conditions, the generation conditions or inference request data may be adjusted and the request resubmitted.

[0126] (S1104) The character generation unit 412 may acquire a character image and send it to the storage control unit 419 for storage. The character generation unit 412 may also read and acquire an adopted character image from the storage control unit 419 based on the identification information of the character image included in the generation conditions. If no character image has been generated or has not been adopted, the character generation unit 412 may be controlled to generate a character image by the process shown in Figure 12.

[0127] (S1105) The lip-sync generation unit 413 may generate mouth movements synchronized with the spoken audio and send them to the storage control unit 419 for storage. The lip-sync generation unit 413 may send a mouth movement generation request to the model linkage unit 418 based on the spoken audio and (if any) the subtitle timestamp sequence. The model linkage unit 418 may generate and send inference request data (e.g., phoneme sequence extraction conditions, frame rate, output format, etc.) to the model server device 130 based on the mouth movement generation request. The model linkage unit 418 may receive mouth movement data (e.g., mouth shape parameter sequence or mask video) from the model server device 130, analyze the received results, format them into a predetermined format, associate them with information identifying the target section, and hand them over to the lip-sync generation unit 413.

[0128] (S1106) The section video generation unit 415 may send a request to the model linkage unit 418 to synthesize the speech video and motion video to generate a section video. The model linkage unit 418 may generate inference request data for the model server device 130 based on the request. The inference request data may include at least information identifying the target section, a background scene image or background scene video (or reference information thereof), a motion video (or reference information thereof), a character image (or reference information thereof), mouth movement data (or reference information thereof), spoken audio (or reference information thereof), and synthesis conditions (e.g., layer order, arrangement, transparency, or presence or absence of subtitles). The model linkage unit 418 may transmit the generated inference request data to the model server device 130. If the received result does not meet predetermined format conditions (e.g., no missing essential materials, time length consistency, or codec consistency), the synthesis conditions or inference request data may be adjusted and a re-request may be made.

[0129] (S1107) The model linkage unit 418 receives section videos from the model server device 130. The model linkage unit 418 may analyze the received section videos, format them into a predetermined video format (e.g., resolution, frame rate, codec, presence or absence of audio), and pass them to the section video generation unit 415, associating them with information that identifies the target section. Furthermore, the model linkage unit 418 may extract metadata such as the duration of the received section videos and store them so that they can be used for subsequent linkage order information or timestamp recalculation.

[0130] The model linkage unit 418 generates reference information (for example, an object identifier on the storage device 140 or an access URL) linked to the target section identification information for the received section video, saves it to the storage control unit 419, and also hands over the reference information and generation status to the UI data generation unit 407. Based on the handed-over reference information, the UI data generation unit 407 sends screen data to the client device 110 that can be used to instruct it to acquire or play a section video preview.

[0131] (S1108) The section video generation unit 415 sends the section video to the storage control unit 419 for storage.

[0132] (S1109) The UI data generation unit 407 may send screen data indicating that generation is in progress or that a preview is available to the client device 110.

[0133] (S1110) The UI data generation unit 407 may detect the completion of generation for all sections and send screen data with the merge start operation area (combination button 660) activated to the client device 110.

[0134] (S1111) The generation condition receiving unit 402 receives a coupling start operation from the client device 110. The start operation for joining received in (S1111) is an example of a joining instruction, and the joining unit 416 executes the joining process in (S1112) based on the joining instruction. In other words, if no joining instruction has been received, even if each section video has been generated, the joining process as a completed video may not be executed. On the other hand, even after a joining instruction has been received, the joining process may be performed as a sequential update or asynchronous execution, and the reference information of the joining result (completed video or provisional video) may be reflected on the screen later.

[0135] (S1112) The linking unit 416 links multiple section videos according to the section order to generate a completed video. The linking unit 416 may also add background music (BGM) based on the BGM selection information included in the generation conditions. In generating the final video, the connecting unit 416 generates the final video by combining multiple section videos generated for each section according to the section order, and may add background music based on the background music selection information included in the generation conditions. The concatenation unit 416 may apply a transition such as a crossfade between adjacent section videos if a transition type is specified. Furthermore, it may perform mixing of the narrated audio and background music, as well as volume normalization, for the audio of the completed video. The concatenation unit 416 may also encode and generate the completed video in a predetermined container format (e.g., MP4).

[0136] (S1113) The output control unit 417 may send the completed video to the client device 110 or save it to the storage device 140 for output. The output control unit 417 performs at least one of the following as output of the completed video: (i) sending completed video reference information (such as a download or streaming playback URL) to the client device 110, (ii) notifying the storage device 140 that saving is complete, and (iii) issuing a sharing identifier as needed. The client device 110 accepts playback, saving, or sharing operations of the completed video based on the received reference information.

[0137] Figure 12 is a flowchart illustrating an example of a process for generating character images based on human images and making them available for use in video generation. Each process in Figure 12 is an example of an operation performed when the registration of human images and the generation of character images are instructed in an embodiment that uses character images. In embodiments that do not use character images, or in embodiments that select and use pre-prepared character images, at least a portion of (S1201) to (S1210) related to the reception of human images and the generation of character images may be omitted. Furthermore, if only human images are registered and the generation start operation is not performed, (S1204) and subsequent steps will not be executed. In addition, in embodiments that reuse already generated character images, the generation processes in (S1205) and (S1206) may be omitted, and reference information of saved character images may be used.

[0138] (S1201) The UI data generation unit 407 transmits screen data related to the registration of person images and the generation of character images to the client device 110. In this case, the screen data transmitted by the UI data generation unit 407 regarding the registration of a person image and the generation of a character image may include, for example, an input area for a person image (an area that can accept input by file selection or drag and drop), a preview display area for a person image, a deletion operation area for a person image, a character image generation start operation area, a display area indicating the generation status, a preview display area for the generated character image, a character image adoption operation area, and a transition operation area that returns to video generation using the adopted character image. Furthermore, the UI data generation unit 407 may deactivate the generation start operation area when no person image has been input.

[0139] (S1202) The input receiving unit 401 receives person image data from the client device 110. The input receiving unit 401 may also send the person image data to the storage control unit 419 for storage.

[0140] (S1203) The UI data generation unit 407 may also display a screen status to the client device 110 indicating that the reception of the person image data is complete.

[0141] (S1204) The generation condition receiving unit 402 receives a character image generation start operation from the client device 110.

[0142] (S1205) The character generation unit 412 acquires human image data and sends a character image generation request to the model linkage unit 418 to generate a character image based on the human image data. The character image generation request may include the human image data or the feature quantities of the human image data, as well as the specification of the generation style.

[0143] (S1206) The model linkage unit 418 sends a character image generation request to the model server device 130 and receives the character image. The model linkage unit 418 may analyze the received character image, format it into a predetermined format (e.g., image format, resolution, transparency, or color mode), and pass it to the character generation unit 412 in association with the identification information included in the generation request. If the received result does not meet predetermined conditions (e.g., lower resolution limit, degree of distortion, or whether the face region can be detected), the model linkage unit 418 may adjust the generation style or inference request data and make a new request.

[0144] (S1207) The character generation unit 412 sends the character image to the storage control unit 419 for storage.

[0145] (S1208) The UI data generation unit 407 generates a screen state that allows a preview of the character image to be displayed and sends it to the client device 110.

[0146] (S1209) The generation condition receiving unit 402 may receive a character image adoption operation from the client device 110 and store the identification information of the adopted character image as a generation condition. The generation condition receiving unit 402 may also transmit the identification information of the adopted character image to the storage control unit 419 for storage.

[0147] (S1210) The section video generation unit 415 may acquire the selected character image and use it for section video generation.

[0148] (S1211) The storage control unit 419 may manage the person image data as temporary data and delete the person image data after the character image generation is complete or after the adoption is complete.

[0149] According to this embodiment, based on the natural language text data received by the input receiving unit 401, the sub-scenario generation unit 405 generates multiple sub-scenarios corresponding to each of the multiple sections, the section video generation unit 415 generates multiple section videos based on each sub-scenario, the linking unit 416 links the multiple section videos to generate a scenario video (final video), and the output control unit 417 outputs the scenario video. This makes it easier for users to obtain a final video with the intended content organized into sections without having to perform complicated time-axis-centered operations such as timeline editing.

[0150] Furthermore, a generation AI (e.g., model server device 120) can be used to generate an overall scenario from text data, and the overall scenario can be divided into sections by the section division unit 404 to generate sub-scenarios. This makes it easy to create scripts in section units while maintaining the overall structure, even for long text inputs.

[0151] Furthermore, the AI ​​can be used to directly generate sub-scenarios from text data. This makes it easier to quickly obtain a script with the desired number of sections without having to go through the process of generating and dividing the entire scenario.

[0152] Furthermore, the section count control unit 406 controls the generation conditions according to the number of sections received by the generation condition reception unit 402, thereby generating a number of sub-scenarios corresponding to the number of sections. This makes it easier to adjust the video structure to match the user's intended length and configuration.

[0153] Furthermore, the UI data generation unit 407 can output a screen containing multiple section cards and display sub-scenarios corresponding to each section card. This allows the completed video to be created while understanding it as section-based cards, making content verification easier.

[0154] Furthermore, the sub-scenario editing reception unit 408 allows for the editing of sub-scenarios on the section card. This makes it easy to modify the generated results section by section to match the user's intentions.

[0155] Furthermore, the speech synthesis unit 411 generates spoken audio of the sub-scenario being read aloud, and the section video generation unit 415 can generate a section video that includes this spoken audio. This makes it easier to consistently generate everything from text input to videos with audio.

[0156] Furthermore, the voice selection reception unit 409 accepts voice selection operations via the screen and can generate spoken audio based on the selected voice. This makes it easier for users to select and reflect the tone of the speaker in the video.

[0157] Furthermore, the audio data management unit 410 can register user-uploaded audio and make it available for selection on the screen. This makes it possible to generate videos using user-specific speakers, such as the voice quality of a particular person.

[0158] Furthermore, the generation condition receiving unit 402 can receive a merging instruction via the screen, and the linking unit 416 can link multiple section videos based on the merging instruction to generate a scenario video. This allows the user to control the timing of the final video creation after the section has been generated and confirmed.

[0159] Furthermore, the system can generate section videos that include character images generated by the character generation unit 412, and the lip-sync generation unit 413 can be configured to make the character images' mouths move in response to the spoken audio. This makes it easy to generate videos that include speaker expressions.

[0160] Furthermore, it can generate character images based on photos entered by the user. This makes it easier to reflect the user's personal characteristics in the video.

[0161] Furthermore, the background scene generation unit 414 can generate background scene animations corresponding to sub-scenarios and include them in the section animation. This makes it easier to add background expressions that are appropriate to the content of each section.

[0162] <Variation> (Variation 1: Method for generating sub-scenarios) The sub-scenario generation unit 405 may switch between (a) generating an overall scenario from text data and dividing the overall scenario by the section division unit 404 to obtain multiple sub-scenarios, and (b) directly generating multiple sub-scenarios from text data, depending on the generation conditions or processing load.

[0163] (Modification 2: Criteria for section division) The section division unit 404 is not limited to dividing according to the number of sections entered by the operator, but may also divide according to the target length (the total length of the completed video in seconds, the length of each section), according to semantic divisions such as introduction, events, and conclusion, or according to the chapter structure defined in the template.

[0164] (Modification 3: UI equivalent to a section card) The section cards output by the UI data generation unit 407 are not limited to card-shaped displays, but may be implemented as any user interface capable of displaying sub-scenarios for each section, such as list displays, tab displays, accordion displays, or panel displays for each section.

[0165] (Variation 4: Form of editing operation) The sub-scenario editing reception unit 408 is not limited to a mode in which the operator directly edits the sub-scenario text, but may also adopt a mode in which it receives correction instructions and uses a generation AI to regenerate or partially regenerate the sub-scenario to obtain the edited result.

[0166] (Modification 5: Method for generating section videos) In addition to generating and synthesizing multiple elements such as background scene motion images, speech videos, and subtitles, the section video generation unit 415 may also adopt a mode in which it takes a sub-scenario and generation conditions as input to generate a section video with sound in a batch.

[0167] (Modification 6: Generation of spoken audio) The speech synthesis unit 411 is not limited to so-called TTS, and may include a mode in which speaker features are extracted from uploaded audio to generate speech, or a mode in which speech is generated based on speaker features selected from preset audio of multiple speakers. Furthermore, it may also employ a mode in which existing audio data provided by the operator is extracted into sections and added to section videos.

[0168] (Modification 7: Implementation of lip-sync) The lip-sync generation unit 413 is not limited to deforming the mouth region of a 2D character image, but may also control the mouth shape parameters of a 3D model, or generate mouth movements based on a sequence of phonemes or time-stamped subtitles instead of spoken audio.

[0169] (Variation 8: Generation of background scene animation) In addition to directly generating background scene moving images using a generation model, the background scene generation unit 414 may also adopt a method of generating a background still image and then adding effects such as panning, zooming, camera work, and particles to generate a pseudo-moving image.

[0170] (Example 9: Timing of the joining (concatenation) process) The linking unit 416 is not limited to linking all section videos at once after the generation of all section videos is complete, but may also adopt a configuration in which provisional scenario videos are updated sequentially in accordance with the completion of section video generation. In this case, the linking instruction may be implemented as a confirmation instruction or output instruction for the provisional video.

[0171] (Variation 10: Units for assigning background music, etc.) The BGM, sound effects, and transitions accepted by the generation condition acceptance unit 402 are not limited to being applied to the entire completed video; a configuration that accepts and applies different settings for each section may also be adopted.

[0172] (Variation 11: Output format) The output from the output control unit 417 is not limited to making the output downloadable, but may also include saving to the storage device 140, issuing a sharing URL, converting to a predetermined format, or simultaneous output in multiple resolutions and aspect ratios.

[0173] (Variation 12: Arrangement of functions) Each function shown in Figure 4 is not limited to being implemented in the server device 100; it may also be partially implemented in the client device 110 for distributed execution, or it may be executed collaboratively by multiple devices including model server devices 120 and 130.

[0174] (Modification 13: Model linkage configuration) The model linkage unit 418 is not limited to using a single multimodal model, but may also employ a configuration in which language generation, image generation, speech synthesis, video generation, etc., are performed by individual models, or a configuration in which multiple models are switched depending on quality or processing time.

[0175] (Variation 14: Regeneration / alternative generation) If the predetermined quality is not met in each generation process, the server device 100 may adopt a mode of adjusting the prompt or generation conditions to regenerate, or a mode of continuing the process using alternative content based on a template.

[0176] (Variation 15: Event-driven selective execution and asynchronous execution) Each process (each step) shown in Figures 7 to 12 can be selectively executed in response to an on-screen operation event or the occurrence of a predetermined state, and multiple steps can be executed in parallel or asynchronously. For example, if the operator does not select or upload a voice, the corresponding text-to-speech voice generation process may be omitted, and the section video generation process may continue with a default value (silence, or a pre-prepared default speaker voice). Similarly, if the operator does not issue a save instruction for motion, the motion video generation process may be omitted, and the process may continue with a still display or a default slight zoom, etc. Furthermore, if the user edits only a specific section, only the processing corresponding to the edited section will be re-executed, and the processing results for other sections may be reused. Furthermore, each step is not limited to being executed synchronously and sequentially; multiple steps may be executed in parallel, or asynchronously as background processing. In this case, the UI data generation unit 407 may sequentially update and display the progress of each process (generating / generating complete, etc.) on a section-by-section basis, and allow users to preview the completed deliverables sequentially. The linking (combining) by the linking unit 416 may be controlled to be finalized as a completed video or output externally in response to a linking instruction from the operator, or a configuration may be adopted in which the provisional linking result is updated sequentially in response to the completion of section video generation or update.

[0177] The above modifications may be implemented in any combination as long as they do not contradict each other. Furthermore, the above are illustrative examples, and the present invention is not limited to these.

[0178] Although embodiments have been described above, these are presented as examples and are not intended to limit the scope of the invention. Novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0179] 100: Server device 110: Client device 150: Network 210: Control Unit 220: Storage section 230: Communications Department 1000: Information Processing System

Claims

1. An information processing method performed by an information processing system, It accepts text data input in natural language. Multiple sub-scenarios corresponding to each of the multiple sections are generated from the aforementioned text data. Based on each of the aforementioned sub-scenarios, multiple section videos corresponding to each of the aforementioned sub-scenarios are generated. The result of changing the order of the aforementioned multiple sections is received. The linking order information that defines the linking order of the multiple section videos is updated according to the result of the above change, The start and end times associated with the aforementioned multiple section videos are recalculated, A scenario video is generated by linking the multiple section videos according to the updated linking order information. Output the aforementioned scenario video. Information processing methods.

2. The information processing method according to claim 1, The system generates a scenario from the text data using a generation AI, and then generates the multiple sub-scenarios by dividing the scenario into sections. Information processing methods.

3. The information processing method according to claim 1, The generation AI is used to generate the multiple sub-scenarios from the text data. Information processing methods.

4. The information processing method according to claim 1, The number of sections is entered. A number of sub-scenarios corresponding to the number of sections is generated. Information processing methods.

5. The information processing method according to claim 1, The system outputs a screen containing multiple section cards corresponding to each of the aforementioned multiple sections. Each of the aforementioned section cards contains a sub-scenario corresponding to that section card. Information processing methods.

6. The information processing method according to claim 5, Accepts editing operations for the sub-scenario corresponding to the section card, Information processing methods.

7. The information processing method according to claim 1, The system generates a voice-over of the sub-scenario using speech synthesis. To generate the section video including the aforementioned spoken audio, Information processing methods.

8. The information processing method according to claim 7, It accepts voice selection operations via the screen. Based on the selected voice, speech synthesis is used to generate a voice recording that reads out the sub-scenario. Information processing methods.

9. The information processing method according to claim 8, On the aforementioned screen, the audio uploaded by the user is presented as an option to select. Information processing methods.

10. The information processing method according to claim 1, Accepts connection instructions via the screen, Based on the aforementioned linking instructions, the multiple section videos are linked together to generate the scenario video. Information processing methods.

11. The information processing method according to claim 1, At least one of the aforementioned multiple section videos includes a character image. In response to the reading of the sub-scenario corresponding to the section video, the mouth of the character image is made to move. Information processing methods.

12. The information processing method according to claim 11, The character image is generated based on the photograph. Information processing methods.

13. The information processing method according to claim 1, The aforementioned section video includes background scene videos corresponding to the sub-scenario. Information processing methods.

14. The information processing method according to claim 5, The system accepts an operation to change the order of the multiple section cards via the aforementioned screen. In accordance with the above operation, the display order of the multiple section cards corresponding to the multiple sections is rearranged. Information processing methods.

15. The information processing method according to claim 14, In response to a change in the order of the aforementioned multiple section cards, the linking order of the aforementioned multiple section videos is updated. The scenario video is generated according to the updated linking order. Information processing methods.

16. An information processing system, Having one or more control units, The control unit, It accepts text data input in natural language. Multiple sub-scenarios corresponding to each of the multiple sections are generated from the aforementioned text data. Based on each of the aforementioned sub-scenarios, multiple section videos corresponding to each of the aforementioned sub-scenarios are generated. The result of changing the order of the aforementioned multiple sections is received. The linking order information that defines the linking order of the multiple section videos is updated according to the result of the above change, The start and end times associated with the aforementioned multiple section videos are recalculated, A scenario video is generated by linking the multiple section videos according to the updated linking order information. Output the aforementioned scenario video. Information processing system.

17. It is a program, On the computer, A program for causing the information processing method described in any one of claims 1 to 15.

Images