Information processing system, information processing method, and information processing program

Abstract

An information processing system according to the present disclosure includes an acquisition unit that acquires background information related to content production and content information being edited, and a processing unit that outputs assistance information related to the content production based on the background information and the content information being edited.

Description

Technical Field

[0001] This disclosure relates to information processing systems, information processing methods, and information processing procedures. Background Technology

[0002] Techniques for automatically generating video (also known as "motion images") content have been provided. For example, techniques for estimating the three-dimensional pose of a virtual human from a two-dimensional line drawing to generate motion images have been provided (e.g., Patent Document 1).

[0003] Citation List

[0004] [Patent Literature]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2003-058906 Summary of the Invention

[0006] [Technical Issues]

[0007] However, there is room for improvement in conventional techniques. For example, while conventional techniques can generate motion graphics based on a storyboard, they do not provide specific consideration for editing content (such as motion graphics), and there is room for improvement in content creation. Therefore, appropriate support for content creation (such as content editing) is desired.

[0008] Therefore, this disclosure proposes information processing systems, information processing methods, and information processing procedures that can appropriately support content creation.

[0009] [Solution to the problem]

[0010] To address the aforementioned problems, an information processing system according to one embodiment of this disclosure includes: an acquisition unit that acquires background information about content creation and content information being edited; and a processing unit that outputs support information about content creation based on the background information and the content information being edited. Attached Figure Description

[0011] [ Figure 1 [Illustration 1] is a diagram illustrating an example of an information processing system for which this disclosure is made.

[0012] [ Figure 2 [Illustration 1] is a diagram illustrating an example of a hardware configuration related to an information processing system containing the present disclosure.

[0013] [ Figure 3 [] is a diagram showing an example of a user interface.

[0014] [ Figure 4[] is a flowchart illustrating the first processing step.

[0015] [ Figure 5 [] is a diagram showing an example of a user interface.

[0016] [ Figure 6 [] is a diagram showing an example of a user interface.

[0017] [ Figure 7 [] is a diagram showing an example of a user interface.

[0018] [ Figure 8 [Illustration 1] is a diagram showing an overview of the first process.

[0019] [ Figure 9 [] is a diagram illustrating an example of the prompt generation process.

[0020] [ Figure 10 [ ] is a diagram illustrating an example of supporting information generation processing.

[0021] [ Figure 11 [ ] is a flowchart illustrating the second processing procedure.

[0022] [ Figure 12 [ ] is a flowchart illustrating the second processing procedure.

[0023] [ Figure 13 [] is a diagram showing an example of the content.

[0024] [ Figure 14 [ ] is a diagram illustrating an example of supporting information generation processing.

[0025] [ Figure 15 [] is a diagram showing an example of a user interface.

[0026] [ Figure 16 [] is a diagram showing an example of a user interface.

[0027] [ Figure 17 [] is a diagram showing an example of a user interface.

[0028] [ Figure 18 [ ] is a diagram illustrating an example of the timing of intervention.

[0029] [ Figure 19 [ ] is a conceptual diagram illustrating an example of the application of the calm technology.

[0030] [ Figure 20 [Illustration 1] is a hardware configuration diagram illustrating an example of a computer that implements the functions of an information processing device. Detailed Implementation

[0031] In the following, embodiments of the present disclosure will be described in detail based on the accompanying drawings. Note that the information processing system, information processing method, and information processing program according to this application are not limited to these embodiments. In each of the following embodiments, the same components are indicated by the same reference numerals, and redundant descriptions thereof will be omitted.

[0032] The contents of this disclosure will be described in the following order.

[0033] 1. Implementation Method

[0034] 1-1. Overview of the configuration of the information processing system disclosed herein

[0035] 1-2. First Example

[0036] 1-2-1. First Treatment

[0037] 1-2-1-1. The first processing procedure

[0038] 1-2-1-2. Overview of the first treatment

[0039] 1-3. Second Example

[0040] 1-3-1. Second Processing

[0041] 1-3-2-1. The second processing procedure

[0042] 1-3-2-2. Overview of the Second Processing

[0043] 1-4. Other Examples

[0044] 1-5. About AI Models

[0045] 2. Other implementation methods

[0046] 2-1. Other configuration examples

[0047] 2-2. Other

[0048] 3. Hardware Configuration

[0049] <1. Implementation Method>

[0050] <1-1. Overview of the configuration of the information processing system for this disclosure>

[0051] In the following embodiments, an example will be shown where video data (also simply referred to as "video") is generated as content information (also referred to as "content") generated by the information processing system 1. However, the content generated by the information processing system 1 is not limited to video. For example, the content generated by the information processing system 1 can be various types of content, such as animation, music data, and image data. Furthermore, for example, the content generated by the information processing system 1 can be document data, PDF data, presentation (slides) data, etc.

[0052] Figure 1 This is a diagram illustrating an example of an information processing system according to the present disclosure. Information processing system 1 includes a video generation module 100, an information acquisition module 200, a sensor unit 300, and a client UI display unit 400. Although in Figure 1 Only one configuration of each type is shown in the figure, but the information processing system 1 may include multiple video generation modules 100, multiple information acquisition modules 200, multiple sensor units 300 and multiple client UI display units 400.

[0053] First, the configuration of the video generation module 100, which performs video generation processing, will be described. The video generation module 100 includes an information analysis unit 110, a sensor analysis unit 120, a prompt generation unit 130, a video generation unit 140, a sound generation unit 150, a text / signature generation unit 160, a compositing and editing unit 170, an evaluation unit 180, and a client UI module 190, etc.

[0054] Information analysis unit 110 analyzes information. Information analysis unit 110 analyzes input text. For example, information analysis unit 110 analyzes information input from information acquisition module 200. Sensor analysis unit 120 analyzes input sensor information. For example, sensor analysis unit 120 analyzes sensor information acquired from information acquisition module 200.

[0055] The prompt generation unit 130 generates various types of information, such as prompts to be input into an AI (artificial intelligence) model (also referred to simply as a "model"), which is a machine learning model described later. For example, the prompt generation unit 130 generates various types of information needed to generate video (moving images), including prompts to be input into the model. For example, the prompt generation unit 130 generates information to support users performing content creation (also referred to as "support information").

[0056] For example, the prompt generation unit 130 uses user input and pre-stored prompts (e.g., templates) to generate prompts. Note that prompts are merely examples of information input to the AI ​​model (model input information), and the model input information input to the AI ​​model is not limited to prompts and can take any format. The "prompt generation unit" can also be understood as a "model input information generation unit." Figure 1 In the middle, the prompt generation unit 130 includes a scene-oriented generation unit 131, a video-oriented generation unit 132, a sound-oriented generation unit 133, a text / logo-oriented generation unit 134, a text response generation unit 135, an example response generation unit 136, and an example response prompt generation unit 137.

[0057] Scene-oriented generation unit 131 generates various types of information related to scene generation. Scene-oriented generation unit 131 generates input information to be input into a model that outputs a scene. For example, scene-oriented generation unit 131 generates scene data about motion image generation based on an input query. For example, scene-oriented generation unit 131 outputs scene generation information for generating scene data based on an input query.

[0058] The video-oriented generation unit 132 generates various types of information related to video generation. The video-oriented generation unit 132 generates input information to be input into a model that outputs code for configuring 3D (three-dimensional) data. For example, the video-oriented generation unit 132 generates code for configuring 3D data based on scene data. For example, the video-oriented generation unit 132 outputs code generation information used to generate code for configuring 3D data based on scene data.

[0059] The sound-oriented generation unit 133 generates various types of information related to the generation of sound information (audio information). The sound-oriented generation unit 133 generates input information to be input to a model that outputs sound. The text / logo-oriented generation unit 134 generates various types of information related to the generation of text and logos. The text / logo-oriented generation unit 134 generates input information to a model that outputs at least one of text and logo.

[0060] The text response generation unit 135 is a processing unit that outputs support information related to content creation based on background information and content information being edited. The text response generation unit 135 outputs the generated support information. In this way, the information processing system 1 can appropriately support content creation by outputting support information based on background information and content information being edited.

[0061] Background information related to content creation includes at least one of the following: information about the purpose of content creation; scene information about the entire content and the selected clip; the content data itself in the creation process; content information of the selected clip and the preceding / following clips; 3D CG models appearing in the selected clip and their position information; camera equipment operation information, lighting information, sound information, color information, and overlay information used in the selected clip; at least one of the available motion data, background data, character data, and prop data; and at least one of the adjustable settings information.

[0062] Content information refers to information about at least one of video data, animation data, music data, and image data. Content information being edited is data included in the display screen of the content being edited. Support information is suggestive information used to support the editing process of content production. Additionally, for example, support information may be evaluation information about the content information being edited.

[0063] The text response generation unit 135 generates support information about content creation based on background information and the content being edited. The text response generation unit 135 generates support information based on information related to the user performing content creation using the information processing system 1. The text response generation unit 135 generates support information based on information input by the user. The text response generation unit 135 generates support information based on queries from the user related to the content being edited. For example, the text response generation unit 135 generates support information based on a query that includes the user's comments on the content being edited.

[0064] The text response generation unit 135 generates prompts based on user-related information and generates support information based on the prompts. The text response generation unit 135 uses a machine learning agent to generate the support information, which references background information about content creation and information about the content being edited.

[0065] For example, a machine learning agent is an AI model (machine learning model) that outputs supporting information in response to cues, such as model M1, which will be described later. The machine learning agent is an AI model obtained by fine-tuning a pre-trained AI model based on cues related to content creation. For example, the machine learning agent is fine-tuned based on information related to user-generated content. For example, information related to user-generated content includes at least one of previously created content information, user-generated material data, and user-based text data. In this way, information processing system 1 can appropriately support content creation by using the fine-tuned AI model. Model M1, as an example of a machine learning agent, outputs supporting information as text data.

[0066] The text response generation unit 135 generates support information by inputting a prompt generated based on background information into an AI model such as model M1 and causing the AI ​​model such as model M1 to output support information. The text response generation unit 135 also generates support information by inputting a prompt generated based on information about the user who performed the content creation and causing the AI ​​model to output support information. Finally, the text response generation unit 135 generates support information by inputting a prompt generated based on a user's comment and causing the AI ​​model to output support information.

[0067] The example response generation unit 136 outputs examples of the content as supporting information based on background information about content creation and information about the content being edited. For example, the example response generation unit 136 outputs examples of the generated content as supporting information. For example, the example response generation unit 136 generates supporting information by inputting a prompt generated based on background information into an AI model such as model M2 and causing the AI ​​model such as model M2 to output examples of the content.

[0068] Example response prompt generation unit 137 generates prompts to be input into an AI model based on background information about content creation and information about the content being edited. This AI model outputs examples related to the content. For example, example response prompt generation unit 137 outputs the generated prompts as example response prompts. For example, example response prompt generation unit 137 generates example response prompts by inputting prompts generated based on background information into an AI model such as model M3 and causing the AI ​​model such as model M3 to output examples related to the content. Then, example response prompt generation unit 137 generates support information by inputting the generated example response prompts into an AI model such as model M4 and causing the AI ​​model such as model M4 to output examples related to the content.

[0069] The video generation unit 140 performs processing related to video generation. The video generation unit 140 acquires motion image data based on code. The video generation unit 140 uses various types of information generated by the generation unit 130, such as prompts, to generate video. For example, the video generation unit 140 generates motion image data based on code. Note that the video generation unit 140 can acquire motion image data in any manner. For example, the video generation unit 140 can acquire motion image data by sending data for generating motion image data to an external service provider (supplier, etc.) that provides motion image data generation services, and receiving motion image data generated by the service provider from the service provider. Figure 1 In the video generation unit 140, there are a USD generation unit 141, a rendering unit 142, and a video refinement unit 143.

[0070] USD generation unit 141 generates various types of information related to USD (Universal Scene Description). For example, USD generation unit 141 uses AI models such as large language models (hereinafter also referred to as "LLM"), and generates USD-Python using prompts obtained through video prompt generation in video-oriented generation unit 132.

[0071] Rendering unit 142 performs various types of processing related to rendering. Rendering unit 142 performs processing for rendering the USD generated by USD generation unit 141.

[0072] The video refinement unit 143 performs various types of processing for refining the video. The rendering unit 142 improves the quality of the generated video through video refinement processing. For example, the video refinement unit 143 performs image quality improvement processing to improve the image quality of moving image data.

[0073] The sound generation unit 150 performs processing for generating sound (audio). The sound generation unit 150 uses an AI model such as a contrastive learning model and cues obtained through sound cues generated in the sound-oriented generation unit 133 to generate sound information such as BGM (background music), SE (sound effects), narration, and dialogue.

[0074] The text / logo generation unit 160 performs processing for generating at least one of text and a logo. The text / logo generation unit 160 uses information generated by the text / logo generation unit 134 to generate at least one of text and a logo.

[0075] Using the configuration described above, the video generation module 100 generates prompts for scene generation by combining pre-stored prompts and user input. The video generation module 100 generates the scene by inputting the generated prompts into an AI model such as an LLM. Furthermore, the video generation module 100 generates prompts for generating video, audio, and text / signatures based on the generated scene. The video generation module 100 uses the prompts for generating video, audio, and text / signatures, the scene, etc., to perform video generation, audio generation, and text / signature generation.

[0076] The composite editing unit 170 performs editing-related processing. For example, the composite editing unit 170 performs processing to combine (composite) the generated video, audio, and text / logos into a single video.

[0077] Evaluation unit 180 performs evaluation processing for evaluating various targets. Evaluation unit 180 evaluates the information generated by the above configuration. For example, evaluation unit 180 generates information indicating the evaluation of at least one of scene data and motion image data. For example, evaluation unit 180 generates information indicating the evaluation of content information being edited.

[0078] The client-side UI module 190 performs processing related to the output on the client-side UI (user interface). For example, the client-side UI module 190 generates various types of information related to the output on the client-side UI. In this case, the client-side UI module 190 performs processing for generating the UI displayed on the user side. The client-side UI module 190 generates various types of information to be displayed on the client-side UI display unit 400.

[0079] In addition, the information acquisition module 200 acquires various types of information. The information acquisition module 200 includes an information acquisition unit 210, a sensor acquisition unit 220, etc. The information acquisition unit 210 is an information acquisition unit. For example, the information acquisition unit 210 acquires information for processing from the storage unit (memory / storage device 14, etc.) of the information processing system 1. The information acquisition unit 210 acquires background information about content creation. The information acquisition unit 210 acquires information about the content being edited. The information acquisition unit 210 acquires text information input via the keyboard 320 or microphone 330. For example, the information acquisition unit 210 acquires text information input by the user using the keyboard 320 or microphone 330. For example, the information acquisition unit 210 acquires input queries from the user regarding motion image generation.

[0080] Sensor acquisition unit 220 acquires information (also referred to as "sensor information") detected by sensors (such as camera device 340 and motion capture). Information acquisition module 200 provides (sends) the acquired information of various types to video generation module 100. Note that information acquisition module 200 may be integrated with video generation module 100.

[0081] Sensor unit 300 includes various sensors. Sensor unit 300 senses user input. Sensor unit 300 accepts user operations. For example, sensor unit 300 is a receiving unit that accepts operations from the user regarding motion image editing. For example, sensor unit 300 includes a mouse 310, keyboard 320, microphone 330, camera device 340, IMU 350 as an inertial measurement unit, etc. In this way, in addition to mouse 310 and keyboard 320, sensor unit 300 also includes a user terminal (smartphone, etc.) equipped with microphone 330, camera device 340 and IMU 350, sensors such as motion capture sensors, and senses user input.

[0082] The client-side UI display unit 400 displays various types of information to be presented to the client (user). The client-side UI display unit 400 displays the UI generated by the client-side UI module 190 on the client's display screen (display device). For example, the client-side UI display unit 400 is a display control unit that displays a storyboard based on scene data. The storyboard is configured to display motion picture data for each clip of motion graphics.

[0083] Information processing system 1 can have the following functions: Figure 2 The hardware configuration shown. Figure 2 This is a diagram illustrating an example of a hardware configuration related to the information processing system described in this disclosure. Figure 2 In this system, the information processing system 1 includes a cloud-side computer 10 and a client-side computer 20 as hardware configurations. Furthermore, the information processing system 1 may include an information providing device (computer) that provides information resources 30, such as learning data, and AI models 40 to the computer 10.

[0084] Notice, Figure 2 The hardware configuration shown is merely an example, and the information processing system 1 can employ any hardware configuration, as long as it is capable of performing the desired processing. For example, computer 10 and computer 20 can be integrated. Furthermore, information resources 30 and AI models 40 can be stored internally within computer 10.

[0085] Computer 10 includes a CPU (Central Processing Unit) 11, a GPU (Graphics Processing Unit) 12, a communication device 13, and a memory / storage device 14. For example, computer 10 corresponds to... Figure 1The computer 10 includes a video generation module 100 and an information acquisition module 200. The computer 10 may be a service provider (server device) that provides video generation services. The CPU 11 and GPU 12 are so-called processors and perform computational processing (arithmetic processing) related to various types of processing, such as video generation.

[0086] The above are merely examples, and computer 10 can be configured arbitrarily, as long as it can perform the desired processing. For example, computer 10 can perform computational processing (arithmetic processing) related to various types of processing, such as video display, through circuits such as ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays). Furthermore, instead of storing the program in memory (memory / storage device 14, etc.), computer 10 can be configured to have the program directly integrated into the processor's circuitry. In this case, the processor performs its function by reading and executing the program integrated into the circuitry. Each processor in this embodiment is not limited to being configured as a single circuit for each processor; its function can also be achieved by combining multiple independent circuits into a single processor. Furthermore, like computer 10, computer 20 can be configured arbitrarily, as long as it can perform the desired processing.

[0087] Communication device 13 is a communication device having communication functions for sending and receiving information to and from computer 20, information providing device, etc., and may be, for example, a communication circuit, NIC (Network Interface Card), etc. Communication device 13 communicates with other devices such as computer 20 and information providing device via a predetermined network (Internet, etc.). For example, communication device 13 is connected to the predetermined network via wired or wireless means, and sends and receives information to and from other devices such as computer 20 and information providing device.

[0088] The memory / storage device 14 is a storage device for storing various types of information. The memory / storage device 14 is, for example, a semiconductor storage element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk. The memory / storage device 14 stores various types of information for processing by processors such as CPU 11 and GPU 12. The memory / storage device 14 can store information resources 30, AI models 40, etc.

[0089] Computer 20 includes a CPU 21, a GPU 22, a communication device 23, and a memory / storage device 24. For example, computer 20 corresponds to... Figure 1The client-side UI display unit 400 is included. The computer 20 can be a terminal device (portable device, such as a PC (personal computer) or smartphone) used by a user of the video generation service. The CPU 21 and GPU 22 are so-called processors and perform computational processing (arithmetic processing) related to various types of processing such as video display.

[0090] Communication device 23 is a communication device having communication functions for sending and receiving information to and from computer 10, and may be, for example, a communication circuit, NIC, etc. Communication device 23 communicates with other devices such as computer 10 via a predetermined network (Internet, etc.). For example, communication device 23 is connected to the predetermined network in a wired or wireless manner, and sends and receives information to and from other devices such as computer 10.

[0091] The memory / storage device 24 is a storage device for storing various types of information. The memory / storage device 24 is, for example, a semiconductor storage element such as RAM or flash memory, or a storage device such as a hard disk or optical disk. The memory / storage device 24 stores various types of information for processing by processors such as CPU 21 and GPU 22.

[0092] Furthermore, information resource 30 includes various types of information, such as learning data. For example, information resource 30 includes learning data used to train various AI models, such as LLM. AI model 40 includes information about AI models (e.g., LLM) used for processing related to video generation. For example, AI model 40 includes information about various AI models, such as models M1 to M3, which will be described later.

[0093] As described above, information processing system 1 can adopt... Figure 2 Configurations other than those shown. For example, information processing system 1 may have configurations other than those shown. Figure 1 The hardware configuration (also known as "sensor device") corresponding to sensor unit 300 in the middle.

[0094] The sensor device senses user input. The sensor device accepts user actions. Furthermore, for example, the computer 20 may have an I / O interface as an input / output interface device. In this case, the computer 20 receives input from the sensor device via the I / O interface. For example, the computer 20 receives input from an input device such as a keyboard or mouse via the I / O interface. Additionally, the computer 20 outputs information from a display (display device) and speakers (audio output device) via the I / O interface. For example, the computer 20 uses the display and speakers to reproduce video via the I / O interface.

[0095] <1-2. First Example>

[0096] From this point onward, a first example will be described, which is an example of processing performed by information processing system 1. Note that, depending on the device configuration included in information processing system 1, any device capable of performing processing can perform the processing described relative to information processing system 1, depending on the object being processed. First, before describing the processing performed by information processing system 1, Figure 3 The image shows an example of a user interface (UI) used to provide support information to users. Note that descriptions of points similar to those described above will be omitted as appropriate. Figure 3 This is a diagram showing an example of a user interface.

[0097] like Figure 3 As shown, the information processing system 1 provides content CT11 to the user. Content CT11 is a display screen created by the user using information processing system 1 to process content such as video data.

[0098] For example, the information processing system 1 accepts user input information via a display screen such as content CT11. For example, the client UI display unit 400 displays a display screen such as content CT11. Content CT11 shows a state where, for user-generated content—video MV11, a comment CM12 including supporting information generated by the information processing system 1 is displayed. Figure 3 This shows the status of the clip selected by the user in video MV11. Furthermore, in Figure 3 In the text, the comment list TH11, including comment CM12, shows the comment list corresponding to the portion (location) in video MV11 that is attached with icon IC11. For example, comment list TH11 is a chat thread (also simply referred to as a "thread") provided by the chat function.

[0099] Comment CM12 is a suggestion message used to support editing of video MV11 by a fictional creator (hereinafter referred to as "AI creator") who has been given a predetermined personality as an AI co-creator. In the comment list TH11, the comments of creators whose names are preceded by "[AI]" (i.e., comments CM12 and CM14) correspond to the comments of AI creators.

[0100] The process of giving AI creators a personality will be described later, but for example, it can be done by adding information corresponding to the AI ​​creator's personality (also known as "personality information") to the prompt input of the AI ​​model. Note that giving AI creators a personality is not limited to adding personality information to the prompt, and can be done in any way possible, as long as the desired personality can be given. For example, a personality can be given to an AI creator by using an AI model corresponding to each individual AI creator.

[0101] therefore, Figure 3 The illustration shows a scenario where "Toru Kikuchi," an AI creator endowed with a predetermined personality, provides suggestions to a user (hereinafter referred to as the "target user") editing the video MV11. Figure 3 In this context, it is represented as "User X".

[0102] <1-2-1. First Treatment>

[0103] <1-2-1-1. The First Processing Step>

[0104] From this point onward, an example of information processing (also referred to as "first processing") performed by the information processing system 1 in the first example will be described. First, an example will be used... Figure 4 The process of the first treatment is described. Figure 4 This is a flowchart illustrating the first processing step.

[0105] exist Figure 4 In the first process shown, in the information processing system 1, the user selects a time and position within the motion image from the timeline to begin commenting (step S101). For example, as... Figure 5 As shown, a target user using information processing system 1 to edit video MV11 selects any position in video MV11 and begins commenting. For example, in information processing system 1, the user selects the position to comment on and begins commenting by performing a menu selection such as right-clicking.

[0106] In information processing system 1, a user makes a comment (step S102). For example, such as... Figure 5 As shown, the target user using the information processing system 1 to edit the video MV11 selects the location in the video MV11 with the attached icon IC11 and makes a comment. Figure 5 This is a diagram illustrating an example of a user interface. (For example...) Figure 5 As shown, information processing system 1 provides content CT12 to the user. Content CT12 is a display screen showing the state after receiving the first comment from the target user (i.e., the state before content CT11).

[0107] Content CT12 shows a state where the target user editing video MV11 has selected a location in video MV11 marked with icon IC11, and a comment list TH11, including entered comments CM11, is displayed. Figure 5In this context, the target user mentions the AI ​​creator "Toru Kikuchi" and enters comment CM11, which is a question such as "Should I make this person's movements more natural?". In this way, the target user comments by mentioning the AI ​​creator "Toru Kikuchi". The target user enters comment CM11 by typing it into the input field located at the bottom of the comment list TH11.

[0108] Therefore, the information processing system 1 accepts comments CM11 from the user mentioning the AI ​​creator. Note that the information processing system 1 can accept user comments in any manner. For example, the methods of accepting input from the user are not limited to those described above, and the information processing system 1 can accept input from the user's comments via voice input. Furthermore, for example, the information processing system 1 can accept user comments by having the user specify a specific time and correction position on the timeline and ask questions in text form.

[0109] exist Figure 6 In the middle, add an at symbol at the beginning of the creator's name ( The status indicator (symbol) refers to the status of the creator corresponding to the creator's name, but this reference is not limited to adding the @ symbol and can be performed in any manner. Note that the information processing system 1 waits until the user makes a comment (step S102). That is, as a process of the information processing system 1, step S102 can be a conditional branch regarding whether the user's comment has been accepted, and if the user's comment has been accepted, the process of step S103 is executed, and if the user's comment has not been accepted, the system waits until the user's comment is accepted.

[0110] The information processing system 1, which has already received the user's comment, determines whether the user's comment mentions the AI ​​co-creator (step S103). If the user's comment does not mention the AI ​​co-creator (step S103: no), the information processing system 1 returns to step S102 and repeats the process.

[0111] If a user's comment mentions an AI co-creator (step S103: Yes), the AI ​​co-creator, tuned to the designated role, provides a text response based on motion image information (step S104). For example, when a user's comment mentions at least one AI creator, the information processing system 1 generates a text response to the user's comment based on the personality of the mentioned AI creator. Therefore, the information processing system 1 generates a text response based on the AI ​​creator specified by the target user.

[0112] For example, such as Figure 6As shown, the information processing system 1 generates and outputs comment CM12 as a text response to comment CM11 of the target user, based on the personality of the AI ​​creator "ToruKikuchi" mentioned by the target user. Figure 6 This is a diagram illustrating an example of a user interface. (For example...) Figure 6 As shown, the information processing system 1 provides content CT13 to the user. Content CT13 is a display screen in the following state: output (display) of comment CM12, which includes support information provided by the AI ​​creator in response to the target user's first comment.

[0113] Content CT13 displays the following status: A list of comments TH11, including comment CM12, is shown. Comment CM12 includes support information for the target user, reflecting the personality of the AI ​​creator "Toru Kikuchi". Figure 6 In this context, the information processing system 1 outputs support information based on the personality of the AI ​​creator "Toru Kikuchi" by mentioning the target user. The information processing system 1 generates a text response including support information by inputting a prompt generated based on the target user's comment CM11 into model M1 and having model M1 output a text response including support information.

[0114] Comment CM12 includes support information, which is a response from the AI ​​creator "ToruKikuchi" mentioned by the target user. For example, comment CM12 includes support information suggesting assistance in editing video MV11, such as "The movements are a bit stiff. Making them more natural will allow viewers to watch smoothly without noticing." Therefore, information processing system 1 displays comment CM12, which includes support information suggesting assistance in editing video MV11, to the target user. Furthermore, comment CM12 includes information to confirm with the target user whether an example response needs to be generated (also called "example response necessity confirmation information"), such as "Can we try making it once?". Therefore, information processing system 1 confirms with the target user the necessity of generating an example response corresponding to the support information included in comment CM12, which is a suggestion for supporting the editing of video MV11.

[0115] Information processing system 1 determines whether there is a creation request in the question from the user (step S105). If the user does not provide an example response to the creation request (step S105: No), information processing system 1 returns to step S102 and repeats the process.

[0116] If the user has already made a request to create an example response (step S105: Yes), the AI ​​co-creator, tuned to the designated role, provides an example response based on motion image information (step S106), and returns to step S102 to repeat the process. For example, if the information requesting the generation of an example response is included in the user's comment, the information processing system 1 provides an example response to the user's comment. Furthermore, when the information processing system 1 has confirmed to the target user that it is necessary to generate an example response corresponding to the suggested information (along with suggested information to support editing), and if the user has already provided a response to the confirmed request, the information processing system 1 provides an example response.

[0117] For example, such as Figure 3 As shown, in response to comment CM12, which includes sample response necessity confirmation information such as "Can we try making it once?", the target user inputs comment CM13, which includes sample response request such as "Yes, please". Therefore, information processing system 1 accepts the request and generates target user comment CM13, which corresponds to suggestion information for supporting the editing of video MV11.

[0118] Then, Information Processing System 1 performs generation processing to generate a sample response based on the personality of the AI ​​creator "Toru Kikuchi" mentioned by the target user. After the sample response generation is complete, Information Processing System 1 outputs a comment CM14 such as "I made it. Please check it out," and then outputs a list including multiple videos, etc. Figure 7 The example response EA11 is shown. Figure 7 This is a diagram illustrating an example of a user interface. (For example...) Figure 7 As shown, information processing system 1 provides content CT14 to the user. Content CT14 is a display screen in the following state: based on the support information in comment CM12, it outputs (displays) a sample response EA11 including multiple videos edited to have natural rotational movements. For example, information processing system 1 generates sample response EA11 by inputting a prompt generated based on the support information in comment CM12 to model M2 and causing model M2 to output sample response EA11. In this way, information processing system 1 generates sample response when the user requests it. Figure 7 As shown, the example response EA11 is displayed in a separate section from the video MV11. Figure 7 (One side of the video MV11 in the video).

[0119] As described above, information processing system 1 provides support information to the target user that reflects the personality of the AI ​​creator. For example, when the target user mentions the AI ​​creator, information processing system 1 automatically generates a text response. For example, when the target user requests the creation of videos, motion graphics, etc., information processing system 1 generates an example response. For example, when generating a response, information processing system 1 generates an example response based on various types of background information, including motion image information. Note that if the target user mentions other users who are human creators rather than AI creators, information processing system 1 does not generate a text response or example response based on the AI ​​creator's personality.

[0120] <1-2-1-2. Overview of the First Process>

[0121] Will use Figure 8 An overview of the first process, as described above. Figure 8 This is a diagram showing an overview of the first process.

[0122] Information processing system 1 uses data DT11, including user comments; data DT12, including past comments from AI creators; data DT13, including preset prompts; and data DT14, as a dataset, to generate information for prompt generation. For example, data DT11 includes past user comments from the same thread. For example, data DT12 includes past comments from AI creators from the same thread. For example, data DT13 includes data used as a template for prompts to be input into the AI ​​model. For example, data DT14 includes at least one of content creation information BIN11 to BIN19.

[0123] Content creation information BIN11 indicates the purpose of motion graphics creation. For example, content creation information BIN11 indicates the purpose of motion graphics creation for the target user. Content creation information BIN12 is scene information for the entire scene and the selected clip. For example, content creation information BIN12 is scene information for the entire video of the target user performing content creation and the clip selected by the target user.

[0124] Content creation information BIN13 is information about the entire moving image. For example, content creation information BIN13 is video data edited by the target user. Content creation information BIN14 is information about the selected clip and the preceding / following moving images (multiple images). For example, content creation information BIN14 is information about the clip selected by the target user and the clips before and after that clip.

[0125] Content creation information BIN15 contains the name, location, and other information of the 3D CG models appearing within the selected clip. For example, content creation information BIN15 contains the name, location, and other information of the 3D CG model of a person (e.g., a USD file) appearing in the clip selected by the target user. Content creation information BIN16 contains information such as camera setup operation information, lighting, sound, color, and overlay information used in the selected clip. For example, content creation information BIN16 contains camera setup operation information, lighting information, sound information, color information, and overlay information used in the clip selected by the target user.

[0126] Content creation information BIN17 is information about the currently edited scene, such as clips, characters, and sounds. For example, content creation information BIN17 is the information that the target user is editing, such as clips, characters, and sounds. Content creation information BIN18 contains available motion data, background data, character data, prop data, etc. For example, content creation information BIN18 is the motion data, background data, character data, prop data, etc., possessed by information processing system 1.

[0127] Content creation information BIN19 is information about adjustable settings (such as camera operation). For example, content creation information BIN19 is information about adjustable settings (such as camera operation) related to a video created by a target user. Note that content creation information BIN11 through BIN19 are merely examples, and content creation information can include various types of information besides content creation information BIN11 through BIN19. For example, content creation information may include AI creator personality information indicating the AI ​​creator's personality settings.

[0128] Note that at least one piece of content creation information, such as content creation information BIN11 to BIN19, is used as data DT14. For example, information processing system 1 selects the best information from content creation information BIN11 to BIN19 as data DT14 based on the information of the scene being edited. Furthermore, for example, information processing system 1 selects the best information from content creation information BIN11 to BIN19 as data DT14 based on the comment content of the target user. Additionally, for example, information processing system 1 selects the best information from content creation information BIN11 to BIN19 as data DT14 based on the time and physical location of the video selected by the target user.

[0129] Information processing system 1 uses at least one of data DT11 to DT14 to generate a prompt PT11, which is a prompt for text response generation. Information processing system 1 uses at least one of data DT11 to DT14 to generate the prompt PT11 for text response generation, which serves as input to model M1. For example, model M1 is an AI model that outputs information including support information in response to the input of the prompt for text response generation, said support information being suggestions for supporting content editing. Model M1 can be any AI model such as LLM (Large Language Model), as long as it can obtain the desired output in response to the input.

[0130] exist Figure 8 In this process, the information processing system 1 generates a text response OT11 that includes supporting information by inputting a prompt PT11 to the model M1 and causing the model M1 to output a text response OT11. For example, the information processing system 1 generates a text response OT11 that reflects the personality of the AI ​​creator mentioned by the user by inputting a prompt PT11 that includes the AI ​​creator's personality information to the model M1. The information processing system 1 then displays the generated text response OT11. For example, the information processing system 1 places the text response OT11 on a display screen such as content CT11 and displays it on the client UI display unit 400.

[0131] Information processing system 1 uses at least one of data DT11 to DT14 to generate a cue PT12, which is an example response generation cue. Information processing system 1 uses at least one of data DT11 to DT14 to generate the example response generation cue PT12, which serves as input to model M2. For example, model M2 is an AI model for motion generation that outputs example responses, such as video, in response to the example response generation cue input. Model M2 can be any AI model, such as LLM (Large Language Model), as long as it can produce the desired output in response to the input.

[0132] exist Figure 8In this system, information processing system 1 generates an example response OT12, including information such as video, by inputting a prompt PT12 to model M2 and causing model M2 to output an example response OT12. For example, information processing system 1 generates an example response OT12, including video and other information, reflecting the personality of the AI ​​creator mentioned by the user, by inputting a prompt PT12, including comments from an AI creator mentioned by the user, to model M2. Then, information processing system 1 displays the generated example response OT12. For example, information processing system 1 places the example response OT12 on a display screen such as content CT14 and displays it on the client UI display unit 400.

[0133] Information processing system 1 uses at least one of data DT11 to DT14 to generate a prompt PT13, which is a prompt generation prompt for example response generation. Information processing system 1 uses at least one of data DT11 to DT14 to generate the prompt generation prompt PT13, which serves as input to model M3. For example, model M3 is an AI model that outputs a prompt generation prompt for example response in response to the input of the prompt generation prompt. Model M3 can be any AI model such as LLM (Large Language Model), as long as it can produce the desired output in response to the input.

[0134] exist Figure 8 In this process, information processing system 1 generates prompt PT14 by inputting prompt PT13 into model M3 and causing model M3 to output prompt PT14, which is a prompt used to generate an example response. For example, information processing system 1 generates prompt PT14 reflecting the personality of the AI ​​creator mentioned by the user by inputting prompt PT13, which includes AI creator personality information mentioned by the user, into model M3.

[0135] Then, information processing system 1 uses the generated prompt PT14 for example response generation as input to model M4. For example, model M4 is an AI model for motion generation, etc., which outputs example responses such as videos in response to the input of the prompt for example response generation. As model M4, any AI model such as LLM (Large Language Model) can be used, as long as it can obtain the desired output in response to the input. For example, model M4 can be the same AI model as model M2.

[0136] exist Figure 8In this process, the information processing system 1 generates an example response OT13, including information such as video, by inputting a prompt PT14 to the model M4 and causing the model M4 to output an example response OT13. For example, the information processing system 1 generates an example response OT13, including information such as video, by inputting a prompt PT14 reflecting the personality of an AI creator mentioned by a user to the model M4. Then, the information processing system 1 displays the generated example response OT13. For example, the information processing system 1 places the example response OT13 on a display screen as shown in the content CT14 and displays it on the client UI display unit 400.

[0137] Note that when providing a sample response to the user, the information processing system 1 may generate only one of the sample response OT12 and sample response OT13. For example, when providing the sample response OT12 to the user, the information processing system 1 does not need to perform the processing related to the generation of the sample response OT13, such as not generating prompts PT13 and PT14 and the sample response OT13. Similarly, when providing the sample response OT13 to the user, the information processing system 1 does not need to perform the processing related to the generation of the sample response OT12, such as not generating prompt PT12 and the sample response OT12.

[0138] Here, will be used Figure 9 This describes an example of the prompt generation process performed by information processing system 1. Figure 9 This is a diagram illustrating an example of the prompt generation process. For example, Figure 9 This is a diagram illustrating an example of a prompt generation process that reflects the personality of an AI creator. For example, such as... Figure 9 As shown, the information processing system 1 uses first information INF1 and second information INF2 to generate a prompt PT15 that reflects the personality of "Toru Kikuchi" as the AI ​​creator.

[0139] The first information INF1 includes AI creator personality information indicating that "Toru Kikuchi," as the AI ​​creator, is a film director. Furthermore, the first information INF1 includes information about user requests for video reviews, reflecting the personality of "Toru Kikuchi" as the AI ​​creator. Additionally, the first information INF1 includes information requesting comments about the video, reflecting the personality of "Toru Kikuchi" as the AI ​​creator. Furthermore, the first information INF1 includes constraint information indicating character count constraints when making text responses. Furthermore, the first information INF1 includes constraint information indicating the execution of an example response when an example response is indicated, etc. For example, the first information INF1 is a tuning prompt. The information processing system 1 can use the first information INF1 as data DT13 including preset prompts.

[0140] The second information INF2 includes information about copyrighted works such as videos, scripts, scenes, and music created by "Toru Kikuchi" as an AI creator to date. Additionally, the second information INF2 includes information about materials such as articles, PDFs, and presentation (slideshow) data created by "Toru Kikuchi" as an AI creator to date. Furthermore, the second information INF2 includes information about emails and chats conducted by "Toru Kikuchi" as an AI creator to date. Furthermore, the second information INF2 includes historical information such as comment data created by "Toru Kikuchi" as an AI creator in other people's individual projects to date.

[0141] In this way, the second information INF2 is a dataset input as information indicating the personality of the AI ​​creator. Note that the second information INF2 is not limited to the above and includes various types of information related to the personality of the AI ​​creator. For example, the second information INF2 may include name, age, gender, occupation, experience, education background, awards, technical skills, personality traits, work strengths, daily life, career goals, challenges faced, representative works, industry reputation, influential works or directors, current activities, speaking characteristics, examples of speaking style, etc.

[0142] exist Figure 9In this context, information processing system 1 generates prompt PT15 by adding second information INF2 to first information INF1. For example, information processing system 1 generates prompt PT15 as a prompt for text response generation. Note that information processing system 1 generates prompt PT15 as an example response generation prompt. In this case, information processing system 1 generates the example response generation prompt, for example, by changing "Please output a comment for the input comment" in the first line of the constraint of first information INF1 to "Please output an example response for the input comment".

[0143] Next, we will use Figure 10 This describes an example of supporting information generation processing performed by information processing system 1. Figure 10 This is a diagram illustrating examples of supporting information generation processes. For example, Figure 10 This is a diagram illustrating an example of the support information generation process for video clips selected and commented on by users. For example, as... Figure 10 As shown, the information processing system 1 uses the target user's comments CM21 and the dataset DT21, which includes motion image information (such as the purpose and scene information of the motion image), to generate a cue PT21 for the selected motion image (its clip).

[0144] For example, information processing system 1 uses clips selected by the target user ( Figure 10 The second clip image (CN21) in the image is used to generate the prompt PT21, which is generated from the clip image CN21 and the prompt template TP21 (preset prompt). For example, the template TP21 can be selected based on the mention. Although in Figure 10 The explanation is omitted, but PT21 can also include AI creator personality information indicating the AI ​​creator's personality. For example, PT21 is a cues that reflect and are tuned to the AI ​​creator's personality. Furthermore, such as... Figure 9 As shown, the prompt PT21 may include indication information such as specific instructions, constraint information, etc., regarding the output.

[0145] exist Figure 10 In this process, the information processing system 1 generates a text response OT21 that includes supporting information by inputting a prompt PT21 to the model M1 and causing the model M1 to output a text response OT21. For example, the information processing system 1 generates a text response OT21 that includes supporting information as a suggestion for the user's comment by inputting a prompt PT21 containing the user's comment to the model M1. The information processing system 1 then displays the generated text response OT21. For example, the information processing system 1 places the text response OT21 on a display screen such as content CT11 and displays it on the client UI display unit 400.

[0146] <1-3. Second Example>

[0147] The first example described above is merely an example of processing performed by information processing system 1, and information processing system 1 is not limited to the processing shown in the first example, but can perform various types of processing. For example, an AI creator can make a comment without being mentioned (instructed) by the user. An example of this will be described below as a second example. Note that information processing system 1 can perform processing that combines the first and second processes. Furthermore, descriptions of points similar to those described in the first example will be appropriately omitted.

[0148] <1-3-1. Second Processing>

[0149] <1-3-2-1. The Second Processing Step>

[0150] From this point onward, an example of information processing (also referred to as "second processing") performed by information processing system 1 in the second example will be described. First, an example will be used... Figure 11 and Figure 12 To describe the process of the second processing. Figure 11 and Figure 12 This is a flowchart illustrating the process of the second treatment. For example, Figure 11 This is a flowchart illustrating the process of processing AI creator comment outputs that are not based on user mentions (instructions). For example, Figure 12 This is a flowchart illustrating the process of processing AI creator comment output based on user mentions (instructions). Note that descriptions of points similar to the first process described above will be appropriately omitted. For example, parallel execution... Figure 11 The processing and shown Figure 12 The processing shown.

[0151] exist Figure 11 In the process shown, information processing system 1 determines whether it is time to post a comment (step S201). If it is not time to post a comment (step S201: no), information processing system 1 repeats the process of step S201.

[0152] If it is time to post a comment (step S201: Yes), the AI ​​co-creator posts the comment (text response) (step S202), and processing returns to step S201 to repeat the process. For example, if the information processing system 1 determines that there is a point to comment on in the content edited by the user, the information processing system 1 generates a comment based on the AI ​​creator's personality.

[0153] Note that when multiple personalities exist for an AI creator, the information processing system 1 generates comments based on the AI ​​creator's personality selected according to predetermined criteria. For example, when multiple personalities exist for an AI creator, the information processing system 1 generates comments based on the AI ​​creator's personality previously mentioned by the user. For example, the information processing system 1 generates comments that include supporting information reflecting the AI ​​creator's personality.

[0154] Information processing system 1 outputs the generated comment. For example, information processing system 1 displays an icon indicating the location corresponding to the generated comment on the display screen and also displays it on the client UI display unit 400. For example, as Figure 13 As shown, for video MV21, information processing system 1 makes comments on the positions in video MV21 corresponding to the positions where icon IC21 and icon IC22 are attached. Figure 13 This is a diagram illustrating an example of the content.

[0155] like Figure 13 As shown, when there are multiple locations in the content to be commented on, the information processing system 1 can make multiple comments. For example, when the user selects icon IC21 in video MV21, the information processing system 1 displays a list of comments corresponding to icon IC21 on the client UI display unit 400. For example, when the user selects icon IC22 in video MV21, the information processing system 1 displays a list of comments corresponding to icon IC22 on the client UI display unit 400.

[0156] exist Figure 12 In the process shown, in information processing system 1, a user makes a comment (step S301). Note that the user can comment by selecting any location in the content, or by... Figure 11 The process shown is used to comment on the comments made by information processing system 1. Note that in Figure 12 In the process shown, the information processing system 1 waits until the user makes a comment (step S301). That is, as a process of the information processing system 1, step S301 can be a conditional branch regarding whether the user's comment has been accepted, and if the user's comment has been accepted, the process of step S302 is executed, and if the user's comment has not been accepted, the system waits until the user's comment has been accepted.

[0157] The information processing system 1, which receives a user's comment, determines whether the user's comment mentions the AI ​​co-creator (step S302). If the user's comment does not mention the AI ​​co-creator (step S302: no), the information processing system 1 returns to step S301 and repeats the process.

[0158] If a user's comment mentions an AI co-creator (step S302: Yes), the AI ​​co-creator, tuned to the designated role, provides a text response based on motion image information (step S303). For example, when a user's comment mentions at least one AI creator, the information processing system 1 generates a text response to the user's comment based on the personality of the mentioned AI creator. Therefore, the information processing system 1 generates a text response based on the AI ​​creator specified by the target user.

[0159] Information processing system 1 determines whether a creation request exists in the question from the user (step S304). If the user does not provide an example response to the creation request (step S304: No), information processing system 1 returns to step S301 and repeats the process.

[0160] If the user has already made a request to create an example response (step S304: Yes), the AI ​​co-creator, tuned to the designated role, provides an example response based on motion image information (step S305), and returns to step S301 to repeat the process. For example, if the information requesting the generation of an example response is included in the user's comment, the information processing system 1 provides an example response to the user's comment. Furthermore, when the information processing system 1 has confirmed to the target user that it is necessary to generate an example response corresponding to the suggested information (along with suggested information to support editing), and if the user has already provided a response to the confirmed request, the information processing system 1 provides an example response.

[0161] As described above, the information processing system 1 provides the target user with support information reflecting the personality of the AI ​​creator. For example, the information processing system 1 automatically evaluates moving images and outputs comments reflecting the AI ​​creator's personality even when not explicitly mentioned. For example, the information processing system 1 evaluates content such as moving images based on scene information and images. For example, the information processing system 1 evaluates at least one of good and bad parts and displays comments indicating that evaluation at the corresponding location.

[0162] For example, for Figure 13 In the video MV21, the information processing system 1 generates evaluation information, such as, "Since the product is targeted at Japanese female users, including at least Asian performers will generate empathy among the audience." Furthermore, for example, regarding... Figure 13 In video MV21, information processing system 1 generates evaluation information, such as "The natural movement would be to pick up the speaker that was initially placed on the sofa, rather than turning around to pick up the speaker here." For example, information processing system 1 uses model M1 as described above to generate evaluation information. Information processing system 1 generates evaluation information by inputting prompts including information indicating the evaluation target to model M1 and causing model M1 to output evaluation information.

[0163] <1-3-2-2. Overview of the Second Process>

[0164] Apart from the fact that information processing system 1 outputs comments regardless of user comments, the second processing described above is similar to the first processing, so a detailed overview is omitted, but will be described below. Figure 14 The processing shown is as follows. Figure 14 This is a diagram illustrating an example of supporting information generation processing. Figure 14 An example of a prompt is shown when generating evaluation information, etc., as examples of supporting information.

[0165] For example, such as Figure 14 As shown, the information processing system 1 uses comment CM31 and dataset DT31 to generate prompt PT31. The comment CM31 is a preset comment used to indicate the evaluation of the motion picture clip, and the dataset DT31 includes motion picture information, such as the purpose and scene information of the motion picture.

[0166] For example, information processing system 1 uses an image as the clip to be evaluated ( Figure 14 The second clip in the image (CN31) and the template TP31 (preset prompt) are used to generate the prompt PT31. For example, the template TP31 can be selected based on the mention. Although in Figure 14 The explanation is omitted, but PT31 may include AI creator personality information indicating the AI ​​creator's personality, etc. For example, PT31 is a cues that reflect and are tuned to the AI ​​creator's personality. Furthermore, as... Figure 9 As shown, the prompt PT31 may include indication information such as specific instructions, constraint information, etc., regarding the output.

[0167] exist Figure 14 In this process, the information processing system 1 generates a text response OT31 that includes evaluation information by inputting a prompt PT31 to the model M1 and causing the model M1 to output a text response OT31. For example, the information processing system 1 generates a text response OT31 that includes evaluation information indicating the evaluation of the evaluation target by inputting a prompt PT31 that includes the evaluation target to the model M1. Then, the information processing system 1 displays the generated text response OT31. For example, the information processing system 1 places the text response OT31 on a display screen such as content CT11 and displays it on the client UI display unit 400.

[0168] Information processing system 1 outputs comments containing various evaluation information at any given time. For example, information processing system 1 generates and outputs comments containing evaluation information about moving images for each edited screen. For example, in the case of a clip editing screen, information processing system 1 generates and outputs comments containing evaluation information about the clipping of moving images. For example, in the case of a sound editing screen, information processing system 1 generates and outputs comments containing evaluation information about the sound.

[0169] Furthermore, the information processing system 1 generates and outputs comments that include evaluation information indicating whether the moving image matches the purpose. For example, the information processing system 1 generates and outputs comments that include evaluation information indicating whether overall consistency has been achieved. For example, the information processing system 1 generates and outputs comments that include evaluation information indicating whether the quality of the content (e.g., composition, camera operation, lighting, and color) is high.

[0170] Furthermore, the information processing system 1 outputs comments including evaluation information based on the timing of the user's actions (operations). For example, when a user repeatedly watches the same clip, the information processing system 1 determines that the user feels incongruity in the clip, performs evaluation processing, and generates and outputs comments including evaluation information. For example, if the user edits motion graphics themselves instead of based on suggestions (comments) from AI creators, the information processing system 1 performs evaluation processing and generates and outputs comments including evaluation information.

[0171] Furthermore, when a user is watching a specific clip, the information processing system 1 performs evaluation processing on the N clips before / after (N is any integer 1 or greater), and generates and outputs a comment including the evaluation information. Additionally, the information processing system 1 can perform evaluation processing and generate and output a comment including the evaluation information even when the user is not actively interacting with the system. Furthermore, the information processing system 1 can perform evaluation processing and generate and output a comment including the evaluation information even when the user has logged out of the application (software) used for editing content and has not performed any operation.

[0172] Furthermore, when unnatural facial expressions are included in a moving image, the information processing system 1 can perform evaluation processing and generate and output comments that include evaluation information. For example, when an object (such as a character or person) collides with a wall in 3D CG, the information processing system 1 can perform evaluation processing and generate and output comments that include evaluation information.

[0173] Furthermore, the information processing system 1 can have a critical level (evaluation threshold, etc.) and output the critical level (evaluation threshold, etc.) while evaluating. Additionally, the information processing system 1 can represent the critical level by changing the color of the comment or outputting different sounds.

[0174] Furthermore, the information processing system 1 can differentiate the display status (e.g., icon color, size, or shape) of comments that a user has previously confirmed from those that have not been confirmed. For example, the information processing system 1 can make it easier for a user to determine whether they have previously confirmed a comment by making the icon of a comment that the user has previously confirmed brighter in color.

[0175] Furthermore, the information processing system 1 can accept feedback from users regarding evaluation information. In this case, for example, the information processing system 1 can display a button along with evaluation information, which is used to select a good or bad evaluation as indicated by the evaluation information. When the user selects a button, the information processing system 1 can acquire information indicating the user's preferences (tastes) as user preference information. For example, the information processing system 1 can use the acquired user preference information to generate prompts reflecting the user's preferences.

[0176] By using prompts that reflect user preferences in this way, information processing system 1 can provide appropriate information based on the user. For example, information processing system 1 can enable an AI model (e.g., model M1) to learn using the acquired user preference information. For example, information processing system 1 can use the acquired user preference information to fine-tune model M1, etc.

[0177] <1-4. Other Examples>

[0178] The first and second examples described above are merely examples, and the information processing system 1 is not limited to those described above, and can perform various types of information processing and provide arbitrary UIs. Some examples in this regard will be described below.

[0179] For example, such as Figure 15 As shown, the information processing system 1 can perform processing in response to user operations to apply specific portions of the generated motion image to the motion image. Figure 15 This is a diagram illustrating an example of a user interface. For example, Figure 15 This shows a UI example where the motion is simply dragged and pasted to be applied to the main motion image. Note that... Figure 15 The content CT21 shown includes, in addition to motion information MT21 for applying specific portions to moving images, and... Figure 7 The content shown is the same as CT14, and its description will be omitted appropriately.

[0180] Users can replace only a portion of the main motion image by dragging a portion of the motion image (such as a specified motion or lighting) and pasting it onto the main motion image. Figure 15 This illustrates a scenario where a target user applies the motion of the top example response (also referred to as the "application source response") in example response EA11 to video MV11. When the motion information MT21 of the application source response in example response EA11 is dragged and pasted onto video MV11, information processing system 1 applies the motion corresponding to the motion information MT21 of the application source response to video MV11. Therefore, the motion in video MV11 is changed (updated) to the motion corresponding to the motion information MT21 of the application source response.

[0181] Note that the information processing performed by the information processing system 1 can also be used on the actual scene (shooting location, etc.). For example, the information processing system 1 can evaluate the video by acquiring information from the shooting scene and shooting motion images, storyboards, and shooting schedules. This evaluation can be output in the information processing system 1 as a comment during the rush editing of the shoot, or it can be spoken from a voice interface such as a voice service providing device.

[0182] Furthermore, in Information Processing System 1, users can communicate with AI creators via voice instead of chat functions such as comment lists. For example, when prompted to input a comment into the AI ​​model, Information Processing System 1 can input it along with motion image information, or it can input it separately. For example, in Information Processing System 1, when a user inputs a comment or when the user enters the editing screen, processing time can be shortened by first inputting the portion of the comment other than the user's input (also known as "INPUT comment") into the AI ​​model and then only inputting the INPUT comment portion into the AI ​​model later.

[0183] Furthermore, in information processing system 1, learning about the AI ​​creator (its personality) can be performed in any manner. For example, the AI ​​creator (its personality) learns through comments in a project, but may reset data learned through past chats in the middle. For example, the AI ​​creator (its personality) can manage the learned versions through comments and can revert to a stage where the AI ​​creator (its personality) has completed the learning up to the expected comments.

[0184] Furthermore, in information processing system 1, the AI ​​creator's response can be modified based on the user's (human creator's) level of understanding. For example, information processing system 1 can modify the AI ​​model to which the prompts are input based on set values.

[0185] Furthermore, in Information Processing System 1, for comments, not only can a point on the timeline and a location point be selected, but the time change can also be specified. For example, when a user selects a hand position, even if there is a time change, Information Processing System 1 can continue to specify the hand position that changes over time based on the position information of the 3D CG hand skeleton.

[0186] Furthermore, in information processing system 1, comments can not only specify points as locations, but also change the range in the order of point -> finger -> hand -> arm -> body. For example, information processing system 1 can accept such range changes by the user's mouse scrolling. Additionally, for example, information processing system 1 can automatically select pre-segmented (categorized) ranges.

[0187] Furthermore, the information processing system 1 enables all chat threads to be shared not only with all users (all human creators), but also to be used exclusively for communication between a single user (human creator) and the AI ​​creator in a private mode. Additionally, the information processing system 1 provides a function for majority voting on presented example responses between the user (human creator) and the AI ​​creator.

[0188] Furthermore, the UI described above is merely an example, and the information processing system 1 can provide, for example... Figure 16 The UI shown is shown in the image. Figure 16 This is a diagram showing an example of a user interface.

[0189] like Figure 16 As shown, the information processing system 1 provides content CT31 to the user. Content CT31 is a display screen created by the user using information processing system 1 to process content such as video data.

[0190] For example, information processing system 1 accepts user input information via a display screen as shown in content CT31, etc. For example, client UI display unit 400 displays a display screen as shown in content CT31, etc.

[0191] exist Figure 16 In the process, the user selects the motion correction region LN31 in the content CT31 and the activity tab in motion correction in the comment list TH31. The user then comments by mentioning the AI ​​creator "Kenta Nishimura". Therefore, the information processing system 1 responds to the user's comment and generates a prompt (also known as a "prompt PTX") that reflects the personality of "Kenta Nishimura" as the AI ​​creator.

[0192] Then, the information processing system 1 generates a comment CMX that includes supporting information by inputting a prompt PTX to the model M1 and causing the model M1 to output a comment (also called a "comment CMX"). This comment is a text response that includes supporting information. For example, the information processing system 1 generates a comment CMX that includes supporting information reflecting the personality of "Kenta Nishimura" (an AI creator mentioned by the user) by inputting a prompt PTX that includes the personality information of the AI ​​creator "Kenta Nishimura" (an AI creator mentioned by the user) to the model M1. The information processing system 1 then displays the generated comment CMX. For example, the information processing system 1 places the comment CMX on a display screen such as content CT11 and displays it on the client UI display unit 400. In this way, when a comment is made by mentioning the AI ​​creator, supporting information including evaluation information or (suggestion) suggestions is output.

[0193] As described above, the information processing system 1 outputs support information based on the text, image, and video information input by the user so far, or the text, image, and video information generated from them, at the production site of videos, photos, music, etc. (including real-world locations and within PC tools). For example, in response to the user's text input, a personalized (personalized) AI creator generates text responses and change suggestions for videos, photos, music, etc., to improve production quality.

[0194] For example, Information Processing System 1 automatically evaluates productions using personalized (personalized) AI creators, based on text, images, videos, and audio information input by the user so far, or based on text, images, videos, and audio information generated from them, or user actions. For example, Information Processing System 1 leaves comments such as text, photos, and music by specifying a location and time in an indicated section.

[0195] As described above, information processing system 1 provides services for creating videos with co-creators (co-producers), including AI creators. For example, information processing system 1 performs information processing related to the generation of visuals, stories, and motion graphics. For example, information processing system 1 enables users to have co-producers who write and correct prompts on behalf of the user, allowing the user to adjust the desired motion graphics themselves or generate the desired motion graphics through dialogue with the co-producers.

[0196] For example, information processing system 1 enables co-producers, including users, to collaborate on visual creation. For example, in information processing system 1, when creating visuals, co-producers automatically generate stories and motion graphics. For example, in information processing system 1, the stories and motion graphics to be completed are reviewed and corrected together with the co-producers.

[0197] For example, Information Processing System 1 provides motion graphics editing tools (applications) for co-creation with co-producers, including AI creators. For example, in the tools provided by Information Processing System 1, AI creators and users (humans) participate in projects on an equal footing. For example, in Information Processing System 1, AI creators are registered as video directors, bosses, themselves, etc., by personalizing them, and each co-producer performs content creation and commentary.

[0198] For example, in Information Processing System 1, ideas are collected and presented by mentioning a designated co-producer, and the user (he / she) determines the strategy among them. For example, Information Processing System 1 outputs an evaluation of the content. In Information Processing System 1, the user requests another co-producer to generate ideas. For example, in Information Processing System 1, the request can be made via text along with specific correction locations and times.

[0199] For example, in Information Processing System 1, another co-creator can rate a moving image and leave a comment. Note that in Information Processing System 1, even when an AI creator has commented, the sudden appearance of the comment is the same as when another user (human) has commented, and is readily acceptable to users of Information Processing System 1.

[0200] For example, information processing system 1 accepts input of motion graphics creation purpose information, which is information indicating the user's purpose for creating a motion graphics. The motion graphics creation purpose information indicates what kind of motion graphics to create within the text. For example, motion graphics creation purpose information is useful because, without reducing it to the language level, the client cannot understand what is to be created, and neither can the AI ​​creator or another user.

[0201] Motion graphic creation purpose information includes the purpose of the motion graphic, the message to be conveyed in the motion graphic, the target audience, features of the function / service, the length of the motion graphic, aspect ratio, content style (emotional, commercial, advertising, etc.), and visual style (film, animation, etc.). For example, a user may create motion graphic creation purpose information while collaborating with another co-producer.

[0202] Furthermore, the storyboard described above includes information that allows a clear view of scenes, clips, and the overall image; it serves as a page for grasping the overall picture and becomes the center of editing. Additionally, within the storyboard, the flow of clips / scenes can be corrected, and scenes / cuts can be added or removed, rather than focusing on detailed clip content. For example, in Information Processing System 1, users improve the quality of the storyboard while chatting with co-producers or receiving feedback.

[0203] Furthermore, for the clips (scenes) described above, users can review each clip in detail and correct characters, backgrounds, props, camera setups, lighting, etc. For example, in a clip (scene), by using "+" to describe the characters to be fixed and the necessary props, users can arrange objects created / corrected using "characters" and "props." Additionally, content that users want to correct directly within the clip can be corrected through chat with the co-producer. Furthermore, when users make changes that affect the pre / post-editing clips, the co-producer can add evaluation comments. Additionally, users can improve the quality of motion graphics in the clips while chatting with the co-producer or receiving feedback.

[0204] Furthermore, for the roles described above, for example, users may take many photos or use generative AI to create character images. In Information Processing System 1, the output image is a refined image. Additionally, the user improves the character's quality by chatting with co-creators or receiving feedback.

[0205] Furthermore, regarding the props described above, for example, the user takes in 3D CG and takes in props such as products. In information processing system 1, the image to be output is a refined image. Additionally, the user improves the quality of the props while chatting with co-producers or receiving feedback.

[0206] Furthermore, regarding the overlays described above, for example, users can add text, logos, etc., to the motion graphics they create. Additionally, users can improve the quality of their overlays when chatting with co-creators or receiving feedback.

[0207] Furthermore, regarding the sounds described above, users can attach audio information such as background music (BGM), soundtrack (SE), dialogue, and narration to the created motion graphics. For BGM, users adjust the music by determining "genre," "instrument," "mood / theme," and "intensity level" along a timeline. Additionally, users improve sound quality when chatting with co-producers or receiving feedback.

[0208] In addition, the information processing system 1 can provide a UI that allows direct interaction with various objects, including but not limited to co-creators (co-producers) of the AI ​​creator. Figure 17An example of this is shown in the figure. Figure 17 This is a diagram showing an example of a user interface.

[0209] like Figure 17 As shown, the information processing system 1 provides content CT41 to the user. Content CT41 is a display screen created by the user using information processing system 1 to process content such as video data.

[0210] For example, information processing system 1 accepts user input information via a display screen such as content CT41. For example, client UI display unit 400 displays a display screen such as content CT41.

[0211] For example, information processing system 1 can allow the user to interact with people (AIs) associated with video MV41 (e.g., actors appearing in video MV41 within content CT41 or the cameraman who filmed video MV41). In this case, information processing system 1 updates video MV41 based on the interaction between the user and the people (AIs) associated with video MV41.

[0212] For example, when a user selects the icon CR41 indicating the cameraman in content CT41, information processing system 1 accepts input of a comment corresponding to the cameraman (also known as the "AI cameraman") selected by the user's icon CR41. Figure 17 In the process, when a user selects icon CR41, input is accepted for comments on the cameraman (AI cameraman) corresponding to icon CR41. Figure 17 The example illustrates a scenario where a user comments on a cameraman, "Set the camera to a low angle." In response to the user's comment, the information processing system 1 generates a prompt (also known as a "Prompt PTP") that reflects the AI ​​cameraman's personality.

[0213] Then, the information processing system 1 generates a commentary CMP that includes supporting information by inputting a prompt PTP to model M1 and causing model M1 to output a comment (also called a "commentary CMP"). This commentary is a text response that includes supporting information. For example, the information processing system 1 generates a commentary CMP that includes supporting information reflecting the personality of the selected (specified) AI cameraman by inputting a prompt PTP that includes the AI ​​cameraman's personality information selected (specified) by the user to model M1. The information processing system 1 then displays the generated commentary CMP. For example, the information processing system 1 displays the commentary CMP "Low angle, yes. How about leaving space to capture the room space in the background like this?" and the user-input "Set the camera device to a low angle." Furthermore, the information processing system 1 updates the video MV41 to reflect the content of the commentary CMP. Figure 17 In the middle, the information processing system 1 updates the video MV41 to capture a low angle of the room space in the background.

[0214] In this way, when a user selects the cameraman icon CR41 and comments, the information processing system 1 responds to the comment by outputting support information including evaluation information or (suggested) suggestions (such as comment CMP) and updating the video MV41. Therefore, since the user is able to work while communicating with the creators of the video MV41 (also known as "AI personnel"), video generation can be performed with the feeling of actually performing collaborative work.

[0215] For example, when a user selects actor CS41 in video MV41, information processing system 1 accepts input of comments on actor CS41 selected by the user. Figure 17 This shows a scenario where input for comments about actor CS41 is accepted when the user selects actor CS41. Figure 17 This illustrates a user's comment to actor CS41: "Can you make a slightly sad expression?". Therefore, information processing system 1 updates video MV41 to reflect the user's comment, "Can you make a slightly sad expression?". Figure 17 In the process, the information processing system 1 updates the video MV41 with content based on user comments by changing the actor CS41's expression to a sad expression.

[0216] Furthermore, the information processing system 1 can generate prompts (also called "prompts PTS") reflecting the personality of actor CS41 in response to user comments. In this case, the information processing system 1 generates a comment CMS that includes supporting information by inputting the prompt PTS into model M1 and having model M1 output a comment (also called a "comment CMS"). This comment is a text response that includes supporting information. For example, the information processing system 1 generates a comment CMS that includes supporting information reflecting the personality of actor CS41 selected (specified) by the user by inputting the prompt PTS that includes personality information of actor CS41 selected (specified) by the user into model M1. The information processing system 1 then displays the generated comment CMS. For example, the information processing system 1 displays the comment CMS "Is that so?" and the user-input "Can you make a slightly sad expression?".

[0217] In this way, when a user selects actor CS41 and comments, the information processing system 1 makes changes to actor CS41 based on the comments and updates video MV41. Furthermore, the information processing system 1 can output support information (such as a comment CMS) in response to the user's comments, including evaluation information or (suggested) suggestions. Therefore, since the user can work while communicating with the creators of video MV41 (AI personnel), video generation can be performed with the feeling of actually performing collaborative work.

[0218] In addition, content CT41 includes a group of icons CL41 that indicates co-creators (co-producers) including AI creators. Figure 17 The illustration shows a case where icon group CL41 includes four icons, each corresponding to one of the four co-creators (co-producers).

[0219] For example, when a user selects an icon from the icon group CL41, the information processing system 1 accepts input of comments on the co-creators (co-producers) selected by the user. Figure 17 This shows a scenario where, when a user selects the icon at the top of the icon group CL41 for the AI ​​creator named "AAAA" (also known as "AI Creator Z"), input for a comment on AI Creator Z is accepted. Figure 17 This illustrates a scenario where a user comments "What do you think of this person's actions?" to AI creator Z. Therefore, information processing system 1 generates a prompt (also known as a "prompt PTZ") reflecting the personality of AI creator Z in response to the user's comment.

[0220] Then, Information Processing System 1 generates a Comment CMZ that includes supporting information by inputting a cue PTZ to Model M1 and having Model M1 output a comment (also called a "Comment CMZ"). This comment is a textual response that includes supporting information. For example, Information Processing System 1 generates a Comment CMZ that includes supporting information reflecting the personality of AI creator Z mentioned by the user by inputting a cue PTZ to Model M1 that includes AI creator Z's personality information. Information Processing System 1 then displays the generated Comment CMZ. For example, Information Processing System 1 displays the Comment CMZ "If you made the action a little gentler, it would be conveyed more?" and the user-input "What do you think of this person's actions?".

[0221] In this way, when a user selects an AI creator's icon and comments, the information processing system 1 outputs support information, including evaluation information or (suggested) suggestions (such as comment CMZ), based on the AI ​​creator's personality. Therefore, since the user is able to work while recognizing co-creators (co-producers), video generation can be performed as if talking to icons that indicate co-creators (co-producers).

[0222] Furthermore, information processing system 1 can intervene with the user at any time. In this regard, it will use... Figure 18 To describe the example. Figure 18 This is a diagram illustrating an example of intervention timing. For example, information processing system 1 can be based on, as... Figure 18 The timing conditions shown in TL41 are used to determine the timing of intervention with the user.

[0223] For example, generated based on the user's answers to the following questions. Figure 18 Timing condition TL41 in the context of information processing system 1. For example, information processing system 1 presents the following question #1 to a user and generates timing condition TL41 based on the user's response.

[0224] Question #1: “Please list the items to check for determining when to talk to a person. However, you are in a position to review that person’s work and communicate improvements each time. Furthermore, improvements may be repeated many times. Assuming you are AI, how would you calculate the timing for intervention with a human?”

[0225] Figure 18 The timing conditions TL41 in the document include intervention timing conditions based on an understanding of the user's condition and context. Furthermore, Figure 18 The timing condition TL41 in the text includes intervention timing conditions based on the analysis of user behavior patterns. Additionally, Figure 18 The timing conditions for TL41 include intervention timing conditions based on real-time feedback. Furthermore, Figure 18 The timing conditions TL41 include intervention timing conditions based on user preferences and settings. Furthermore, Figure 18 The timing conditions in TL41 include intervention timing conditions based on determinations of urgency and importance.

[0226] Information processing system 1 is based on Figure 18 The timing condition TL41 is used to determine the timing of intervention. For example, information processing system 1 is based on... Figure 18 The timing condition TL41 is used to determine the timing condition in Figure 11 The timing of posting a comment in step S201. Note that... Figure 18 Timing condition TL41 in the example is merely an example, and other conditions may be included in the timing condition. Figure 18 Various conditions related to the timing of intervention, in addition to the intervention conditions shown.

[0227] Furthermore, information processing system 1 can enable AI models such as LLM to solve problems related to technologies such as calm. In this regard, it will utilize... Figure 19 To describe the example. Figure 19 This is a conceptual diagram illustrating an example of the application of the calm technology. For example, information processing system 1 could be based on, for instance... Figure 19 The concept of calm technology is illustrated to perform processing. For example, calm technology here refers to the design of information technology that integrates into the user's life. For example, calm technology is an information technology used to efficiently transmit information while minimizing the user's attention so that the user can naturally participate in the technology. For example, calm technology is a technology that allows the user to use and subsequently achieve effective functionality without being aware of the digital device's presence.

[0228] For example, in Figure 19 In this context, information such as user operation information / current settings in the system, as shown in the first element EL1, is shared with co-creators (joint producers), as shown in the second element EL2. For example, in... Figure 19 In this context, co-creators (co-producers), as shown in Element 2 (EL2), determine whether to intervene. For example, co-creators (co-producers) determine whether to intervene based on information shared from Element 1 (EL1) and the existence of online LLMs that mimic the user, as shown in Element 3 (EL3). Element 3 (EL3) includes information such as a definition of the user's personality (similar to what co-creators consider) and timing biases learned from past user actions (such as whether there is attention to suggestions from co-creators).

[0229] The second element EL2 acquires the results of the intervention (whether there was attention to the suggestion from the co-creator, etc.). The second element EL2 reflects the acquired intervention results (whether there was attention to the suggestion from the co-creator, etc.) in the first element EL1. The second element EL2 updates the settings of the first element EL1 based on the acquired intervention results (whether there was attention to the suggestion from the co-creator, etc.). Therefore, the information processing system 1 can be updated to a state suitable for the user's personality or past behavior. Thus, the information processing system 1 can effectively deliver information while minimizing the user's attention and enabling the user to participate naturally in the technology. That is, the information processing system 1 can achieve effective functionality without the user being aware of the digital device's existence.

[0230] <1-5. About AI Models>

[0231] Note that for each AI model used in each of the processes described above, its internal structure can be arbitrary, as long as it can output the desired information in response to the input, and is not limited to the examples described in each section. Any combination of the AI ​​model's input, output, and internal structure can be used, as long as it can output the desired information.

[0232] The input to an AI model can be text, images, speech, 3D data, or a combination thereof. Furthermore, the output of an AI model can be text, images, speech, 3D data, etc. Note that the inputs and outputs described above are merely examples, and the AI ​​model described above can have any inputs and outputs.

[0233] Furthermore, the internal structure of an AI model can be arbitrary, depending on the combination of inputs and outputs. That is, the internal structure of an AI model can be anything, as long as the desired output can be obtained from the inputs.

[0234] For example, an AI model can have a Transformer-related architecture. For example, an AI model can have a Transformer-related architecture and perform processing considering context such as sequential relationships within the data (e.g., text and time-series data). For example, an AI model can have a self-attention mechanism. For example, an AI model can have arbitrary attention mechanisms, such as single-head attention or multi-head attention. Note that an AI model is not required to have an attention mechanism.

[0235] AI models can have mechanisms for extracting features from input. For example, an AI model can have an encoder. AI models can also have mechanisms for generating information based on the extracted features. For example, an AI model can have a decoder.

[0236] AI models can have structures related to CNNs (Convolutional Neural Networks). For example, when performing image processing, AI models can have CNN-related structures. For example, an AI model can have at least one of convolutional layers, pooling layers, fully connected layers, etc.

[0237] Note that the internal structure described above is merely an example, and the AI ​​model described above can have arbitrary internal structures. For example, the AI ​​model can have skip connections. Furthermore, the AI ​​model can have structures related to diffusion models.

[0238] Furthermore, the AI ​​models described above can be generated (learned) through arbitrary learning processes. An AI model can be a machine learning model learned using any machine learning technique. For example, an AI model can be generated by fine-tuning a so-called base model (foundation model) to apply it to a specific task (e.g., text response generation, example response generation, etc.). For instance, an AI model such as the LLM described above can be generated by fine-tuning a base model to apply it to a specific task.

[0239] The base model here is a model that is learned to be applicable to a variety of tasks (e.g., to be able to perform a variety of tasks). For example, the base model is a neural network pre-trained on a large unlabeled dataset. Note that the base model can have arbitrary architectures, such as Transformer-based architectures. For example, the base model can be generated through self-supervised learning using data without proper labels. As described above, the base model is fine-tuned to suit a wide range of downstream tasks.

[0240] For example, when applied to a text response generation task, the base model is fine-tuned to suit the task, and an AI model (model M1, etc.) is generated for the text response generation task. Furthermore, for example, when applied to an example response generation task, the base model is fine-tuned to suit the example response generation task, and an AI model (model M2, etc.) is generated for the example response generation task.

[0241] For example, an AI model (model M1, etc.) applied to a text response generation task learns using learning data, which includes a combination of input information corresponding to the AI ​​model and text responses (also called "correct information") that serve as the correct output when input information is input. Note that the learning data, such as input information and correct information, can be data created by humans or data automatically generated by a computer that generates the learning data. For example, text responses used as correct information can be data created by humans. The example model M1 will be briefly described below. For example, model M1 learns to output correct information corresponding to each piece of input information in the learning data when input information is input. For example, model M1 learns by adjusting (correcting) parameters (connection coefficients) so that the error between the output of model M1 and the correct information corresponding to the input information when a specific input information is input is reduced using techniques such as backpropagation (error backpropagation method). Furthermore, other AI models, such as AI models (model M2, etc.), applied to the example response generation task, can learn through a similar learning process.

[0242] Note that the learning process described above is merely an example, and the AI ​​model described above can be trained through any learning process depending on its input, output, and internal structure. For example, the AI ​​model can learn using unsupervised learning techniques such as GANs (Generative Adversarial Networks). Furthermore, the AI ​​model can learn in a distributed state without aggregating data, such as through joint learning. In this case, each video generation service device (server, etc.) generates a local model that learns using data collected by that service, and a server (aggregation server) that aggregates the local model information (parameters, etc.) generated by each video generation service device (server, etc.) can use this local model information to generate a global model. In this case, information processing system 1 can receive the global model generated by the aggregation server from the aggregation server and use the received global model as an AI model for processing.

[0243] In this way, the AI ​​models described above can be generated (learned) by any computer. That is, the learning process for generating the AI ​​models can be performed by any device (computer, etc.) of the information processing system 1, or by a device outside the information processing system 1. For example, if at least one of the AI ​​models described above is generated by a device outside the information processing system 1, the information processing system 1 obtains the AI ​​model from the device outside the information processing system 1 and performs processing using the obtained AI model.

[0244] <2. Other Implementation Methods>

[0245] In addition to the above-described embodiments and modifications, the processing according to the embodiments described above can be implemented in various different forms (modifications).

[0246] <2-1. Other Configuration Examples>

[0247] The configuration of the information processing system 1 described above is merely an example, and the functional division within the information processing system 1 can be implemented in any manner. That is, the configuration described above is an example, and the information processing system 1 can have any functional division and any configuration, as long as it can provide services related to the video generation described above. For example, the information processing system 1 can be configured by a device (computer, etc.) performing the processing described above. In this case, the device of the information processing system 1 can have the functions of a video generation module 100, an information acquisition module 200, a sensor unit 300, and a client UI display unit 400. For example, the video generation service provided by the information processing system 1 can also be provided to the user as a program such as a tool (AI-assisted creation tool) that operates on a terminal device (computer 20, etc.) used by the user.

[0248] <2-2. Other>

[0249] Furthermore, in the processes described in the various embodiments above, all or part of the processes described as automatically executed can be performed manually, or all or part of the processes described as manually executed can be performed automatically by known methods. Additionally, unless otherwise stated, the information shown in the above documents and figures, including processing procedures, specific names, and various types of data and parameters, can be arbitrarily changed. For example, the various types of information shown in the figures are not limited to the information shown.

[0250] Furthermore, each component of each illustrated device is conceptual in function and does not necessarily need to be physically configured as shown. That is, the specific form of distribution / integration of the devices is not limited to the form shown, and all or part of them can be configured by distribution / integration in arbitrary units, either functionally or physically, depending on various loads and usage conditions.

[0251] Furthermore, the various implementation methods and modifications described above can be appropriately combined within the scope where the content being processed does not contradict each other.

[0252] Furthermore, the effects described in this specification are merely illustrative and not restrictive, and other effects may exist.

[0253] <3. Hardware Configuration>

[0254] An information processing device (information equipment) having a video generation module 100, an information acquisition module 200, a client UI display unit 400, etc., according to the various embodiments described above, for example, by having such Figure 20 The configuration shown is implemented by computer 1000. A description will be given using an information processing device having a video generation module 100 as an example. Figure 20 This is a hardware configuration diagram illustrating an example of a computer that implements the functions of an information processing device. The computer 1000 includes processing circuitry 1100, RAM 1200, ROM 1300, auxiliary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. Each component of the computer 1000 is connected via a bus 1050.

[0255] The processing circuit 1100 operates based on programs stored in ROM 1300 or auxiliary storage device 1400 and controls each component. For example, the processing circuit 1100 loads programs stored in ROM 1300 or auxiliary storage device 1400 into RAM 1200 and executes processing corresponding to various programs.

[0256] ROM 1300 stores boot programs such as BIOS (Basic Input / Output System) and programs based on the hardware of computer 1000, which are executed by processing circuit 1100 when computer 1000 starts up.

[0257] Auxiliary storage device 1400 is a computer-readable recording medium that non-transitorily records programs executed by processing circuitry 1100 and data used by such programs. Specifically, auxiliary storage device 1400 is a recording medium that records programs used in each process of an information processing apparatus having video generation module 100 according to an embodiment, and the program is an example of program data 1450.

[0258] Communication interface 1500 is an interface for connecting computer 1000 to external network 1550. Communication interface 1500 corresponds to a communication unit (communication device) provided in an information processing apparatus having video generation module 100. For example, processing circuit 1100 receives data from other devices or sends data generated by processing circuit 1100 to other devices via communication interface 1500.

[0259] Input / output interface 1600 is an interface for connecting input / output device 1650 and computer 1000. For example, processing circuitry 1100 receives data from input devices such as microphone 1900 or touch panel via input / output interface 1600. Furthermore, processing circuitry 1100 transmits data to output devices such as display unit 1700 or speaker 2000 via input / output interface 1600. Additionally, input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium. The medium is, for example, optical recording media such as DVD (Digital Universal Disc) and PD (Phase Change Rewritable Disc), magneto-optical recording media such as MO (Magneto-Optical Disc), magnetic tape, magnetic recording media, or semiconductor memory.

[0260] Display unit 1700 is an interface for displaying information processed by computer 1000. Display unit 1700 is, for example, a liquid crystal display or an organic EL display (organic electroluminescent display). Alternatively, display unit 1700 can be a touch panel display device or an image projection device.

[0261] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture voice. The speaker 2000 is an interface for the computer 1000 to output processed voice. Each component of the computer 1000 is connected via a bus 1050. Each interface does not necessarily need to be located inside the computer 1000 and can be located outside the computer 1000 via a network or the like. Furthermore, each component configured in the computer 1000 can be controlled by circuitry different from the processing circuitry 1100. For example, the display unit 1700 may not be controlled by the processing circuitry 1100, but rather by dedicated display processing circuitry located within the display unit 1700.

[0262] For example, when computer 1000 is used as an information processing apparatus having a video generation module 100 according to an embodiment, the processing circuitry 1100 of computer 1000 is used as the video generation module 100 by executing a program loaded on RAM 1200. Furthermore, auxiliary storage device 1400 stores the information processing program according to this disclosure and various types of data stored in the storage units of the information processing apparatus. Note that the processing circuitry 1100 reads and executes program data 1450 from auxiliary storage device 1400; however, as another example, these programs can be obtained from other devices via external network 1550. That is, auxiliary storage device 1400 is not limited to being inside computer 1000 and can be placed outside computer 1000. Note that the processing circuitry 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0263] Note that this technology can also be configured as follows.

[0264] (1) An information processing system, comprising:

[0265] The acquisition unit acquires background information about content creation and information about the content being edited; and

[0266] The processing unit outputs support information about content creation based on background information and the content being edited.

[0267] (2) The information processing system according to (1), wherein the processing unit outputs support information based on information relating to the user who performs content creation using the information processing system.

[0268] (3) The information processing system according to (2), wherein the processing unit outputs support information based on information input by the user.

[0269] (4) The information processing system according to (3), wherein the processing unit outputs support information based on queries from the user relating to the content information being edited.

[0270] (5) An information processing system according to any one of (2) to (4), wherein the processing unit generates prompts based on information related to the user and outputs support information based on the prompts.

[0271] (6) An information processing system according to any one of (1) to (5), wherein the processing unit uses a machine learning agent to output support information, the machine learning agent referring to background information about content creation and content information being edited.

[0272] (7) According to the information processing system of (6), the machine learning agent is a machine learning model that outputs supporting information in response to cue input, and

[0273] The processing unit generates support information by taking input prompts generated based on background information and having the machine learning model output support information.

[0274] (8) The information processing system according to (7), wherein the processing unit generates support information by inputting prompts based on information about the user performing the content creation and causing the machine learning model to output support information.

[0275] (9) The information processing system according to (8), wherein the processing unit generates support information by inputting prompts generated based on user comments and causing the machine learning model to output support information.

[0276] (10) An information processing system according to any one of (6) to (9), wherein the machine learning agent is a machine learning model obtained by fine-tuning a pre-trained machine learning model based on prompts related to content creation.

[0277] (11) The information processing system according to (10), wherein the machine learning agent is fine-tuned based on information related to the creation of content by the user.

[0278] (12) The information processing system according to (11) includes, wherein the information relating to the creation of content by the user includes at least one of the following: content information previously created by the user, material data created by the user, and text data based on the user.

[0279] (13) An information processing system according to any one of (6) to (12), wherein the machine learning agent will support the output of information as text data.

[0280] (14) An information processing system according to any one of (1) to (13), wherein the content information is information relating to at least one of the following: video data, animation, music data and image data.

[0281] (15) An information processing system according to any one of (1) to (14), wherein the background information on content creation includes at least one of the following: information on the purpose of content creation; scene information on the entire content and the selected clip; the content data itself in the creation process; content information of the selected clip and the clips before and after it; 3D CG models appearing in the selected clip and their position information; at least one of the following: camera device operation information, lighting information, sound information, color information and overlay information used in the selected clip; at least one of the following: motion data, background data, character data and prop data; and adjustable settings information.

[0282] (16) An information processing system according to any one of (1) to (15), wherein the content information being edited is data included in the display screen of the content being edited.

[0283] (17) An information processing system according to any one of (1) to (16), wherein the support information is suggestion information used to support the editing of content production.

[0284] (18) An information processing system according to any one of (1) to (17), wherein the support information is evaluation information about the content being edited.

[0285] (19) An information processing method, comprising:

[0286] Obtain background information on content creation and the content being edited; and

[0287] Output support information about content creation based on background information and the content being edited.

[0288] (20) An information processing program that causes a computer to perform:

[0289] Obtain background information on content creation and the content being edited; and

[0290] Output support information about content creation based on background information and the content being edited.

[0291] [List of reference numerals]

[0292] 1. Information Processing System

[0293] 100 Video Generation Module

[0294] 110 Information Analysis Unit

[0295] 120 Sensor Analysis Unit

[0296] 130 Prompt Generation Unit

[0297] 131 Scene-Oriented Generation Unit

[0298] 132 Video-oriented generation units

[0299] 133 Sound-oriented generation unit

[0300] 134 Text / tag generation units

[0301] 135 Text Response Generation Unit

[0302] 136 Example Response Generation Unit

[0303] 137 Example Response Prompt Generation Unit

[0304] 140 video generation units

[0305] 141 USD generation unit

[0306] 142 rendering units

[0307] 143 Video Refinement Units

[0308] 150 sound generation units

[0309] 160 Text / Logo Generation Units

[0310] 170 Composite Editing Units

[0311] 180 Evaluation Units

[0312] 190 Client UI Module

[0313] 200 Information Acquisition Module

[0314] 210 Information Acquisition Unit

[0315] 220 Sensor Acquisition Unit

[0316] 300 sensor units

[0317] 400 Client UI Display Units

Claims

1. An information processing system, comprising: The acquisition unit acquires background information about content creation and information about the content being edited. as well as The processing unit outputs support information about the content creation based on the background information and the content being edited.

2. The information processing system according to claim 1, wherein, The processing unit outputs the support information based on information relating to the user who uses the information processing system to perform the content creation.

3. The information processing system according to claim 2, wherein, The processing unit outputs the support information based on the information input by the user.

4. The information processing system according to claim 3, wherein, The processing unit outputs the support information based on a query from the user relating to the content being edited.

5. The information processing system according to claim 2, wherein, The processing unit generates a prompt based on information related to the user, and outputs the support information based on the prompt.

6. The information processing system according to claim 1, wherein, The processing unit uses a machine learning agent to output the support information, which refers to the background information about content creation and the content being edited.

7. The information processing system according to claim 6, wherein, The machine learning agent is a machine learning model that outputs the supporting information in response to prompted input, and The processing unit generates the support information by inputting a prompt generated based on the background information and causing the machine learning model to output the support information.

8. The information processing system according to claim 7, wherein, The processing unit generates the support information by inputting a prompt based on information about the user who performed the content creation and by having the machine learning model output the support information.

9. The information processing system according to claim 8, wherein, The processing unit generates the support information by inputting a prompt generated based on the user's comment and having the machine learning model output the support information.

10. The information processing system according to claim 6, wherein, The machine learning agent is a machine learning model obtained by fine-tuning a pre-trained machine learning model based on prompts related to content creation.

11. The information processing system according to claim 10, wherein, The machine learning agent is fine-tuned based on information related to user-generated content.

12. The information processing system according to claim 11, wherein, Information relating to the creation of content by the user includes at least one of the following: content information previously created by the user, material data created by the user, and text data based on the user.

13. The information processing system according to claim 6, wherein, The machine learning agent outputs the support information as text data.

14. The information processing system according to claim 1, wherein, The content information is information relating to at least one of the following: video data, animation, music data, and image data.

15. The information processing system according to claim 1, wherein, The background information regarding content creation includes at least one of the following: Information about the purpose of content creation; Information about the overall content and the scene of the selected clip; The content data itself during the creation process; Information about the selected clip and the clips before and after it; The 3D CG model appearing in the selected clip and the position information of the 3D CG model; At least one of the following: camera device operating information, lighting information, sound information, color information, and overlay information used in the selected clip; Possess at least one of the following: motion data, background data, character data, and item data; And adjustable settings.

16. The information processing system according to claim 1, wherein, The content being edited is data included in the display screen of the content being edited.

17. The information processing system according to claim 1, wherein, The support information refers to suggested information used to support editors in creating content.

18. The information processing system according to claim 1, wherein, The support information is evaluation information about the content being edited.

19. An information processing method, comprising: Obtain background information about content creation and the content being edited; as well as Based on the background information and the content being edited, output support information regarding the creation of the content.

20. An information processing program that causes a computer to perform: Obtain background information on content creation and the content being edited; and Based on the background information and the content being edited, output support information regarding the creation of the content.

Images