Information processing device, information processing method, and program

Abstract

Reduce the burden of modifying material to fit your content. [Solution] An information processing device 1 has a receiving unit 131 that receives materials for constituting video content and generation instructions for generating video content, an acquisition unit 132 that acquires metadata, which is data describing the materials, a determination unit 133 that identifies correction content for applying the materials to the video content based on the generation instructions and the material metadata, and a generation unit 134 that generates a prompt for inputting instructions to modify the material to an image generation AI based on the identified correction content.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art There is known a device that creates content by combining materials (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2024-027453 A Summary of the Invention [Problem to be solved by the invention]

[0004] Such materials may not match the desired video content, which limits the creation of content while imposing a heavy burden on the user to manually correct the materials.

[0005] The present invention has been made in consideration of these points, and has an object to reduce the burden of modifying materials to fit content. [Means for solving the problem]

[0006] The information processing device of the first aspect of the present invention has a receiving unit that receives materials for constituting video content and generation instructions for generating the video content, an acquisition unit that acquires metadata which is data describing the materials, an identification unit that identifies modification content for applying the materials to the video content based on the generation instructions and the metadata of the materials, and a generation unit that generates a prompt for inputting instructions to modify the materials to an image generation AI based on the identified modification content.

[0007] The metadata includes an aspect ratio of the material, and the generation instructions include a reference aspect ratio that is a standard aspect ratio for the video content, and when the aspect ratio indicated by the metadata differs from the reference aspect ratio, the identification unit identifies a change in the aspect ratio of the material as the modification content, and the generation unit may generate a prompt to change the aspect ratio of the material to the aspect ratio identified by the identification unit.

[0008] The generation instructions include a description describing the video content, and when the identification unit identifies a change in aspect ratio as the modification content, the generation unit may generate a prompt based on the generation instructions to instruct the user to supplement the missing part of the aspect ratio of the material based on the modification content based on the description describing the video content indicated by the generation instructions.

[0009] When the identification unit identifies a change in aspect ratio as the modification content, the generation unit may generate a prompt based on the generation instruction to instruct the cutting out of any portion that would become redundant if the aspect ratio of the material was changed based on the modification content.

[0010] The material includes a photographic image, the metadata includes a description describing a subject of the photographic image, and the generation instructions include a description describing the video content, and when the description describing the subject of the photographic image indicated by the metadata does not match the description describing the video content indicated by the generation instructions, the generation unit may generate a prompt to modify the photographic image based on the description describing the video content indicated by the generation instructions.

[0011] The apparatus may further include a determination unit that determines whether or not modification is required to apply the material to the video content based on the metadata and the generation instructions, and the identification unit may identify the content of the modification when the determination unit determines that modification of the material is required.

[0012] The generation instruction may further include a restriction indicating matters on which expression in the video content is restricted, and the generation unit may generate a prompt further including the restriction.

[0013] An information processing method of the second aspect of the present invention includes the steps of: receiving materials for constituting video content and generation instructions for generating the video content, executed by a computer; acquiring metadata which is data describing the materials; identifying modification content for applying the materials to the video content based on the generation instructions and the metadata of the materials; and generating a prompt for inputting instructions to modify the materials to an image generation AI based on the identified modification content.

[0014] The computer is caused to execute the steps of accepting materials for constituting video content and generation instructions for generating the video content, acquiring metadata which is data describing the materials, identifying modifications for applying the materials to the video content based on the generation instructions and the metadata of the materials, and generating a prompt for inputting instructions to modify the materials to an image generation AI based on the identified modifications. Effect of the Invention

[0015] The present invention has the effect of reducing the burden of modifying materials to fit content. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram for explaining an overview of an information processing device 1 according to an embodiment. [Diagram 2] 1 is a block diagram showing a configuration of an information processing device 1. FIG. [Diagram 3] 13 is a diagram showing an example of a reception screen displayed by a reception unit 131. FIG. [Figure 4]FIG. 11 is a diagram illustrating an example of a process performed by a specification unit 133. [Diagram 5] 3 is a flowchart showing a process flow in the information processing device 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] [Outline of Information Processing System S] FIG. 1 is a diagram for explaining an overview of an information processing device 1 according to an embodiment. The information processing system S is a system for providing video content. The information processing system S generates video content based on materials specified by a user. The information processing system S has an information processing device 1, an information terminal 2, and a generation device 3. The materials are contents that are elements that make up the video content. The materials are video data, still image data, text data, audio data, music data, etc.

[0018] The information processing device 1 is a device for generating a prompt for giving instructions to a machine learning model. The information processing device 1 determines whether or not the material needs to be modified based on the content specified by the user and the metadata of the material specified by the user. If modification is necessary, the information processing device 1 generates a prompt for instructing the generation device 3 to modify the material.

[0019] The information terminal 2 is a terminal used by a user of the information processing system S. The information terminal 2 is, for example, a smartphone, a tablet, or a personal computer. The information terminal 2 transmits to the information processing device 1 an instruction regarding the details of the content to be generated.

[0020] The generating device 3 generates video content, still images, etc. based on the received instructions. The generating device 3 has a generative model (hereinafter, sometimes referred to as "image generation AI") that is a trained model trained to generate video or still images based on instructions written in natural language, and outputs the video or still images generated by the generative model based on the input instructions.

[0021] The processing of the information processing device 1 will be described. The information processing device 1 acquires materials and a generation instruction from the information terminal 2 ((1) in FIG. 1). The generation instruction is an instruction to generate video content. As an example, the generation instruction includes a description explaining the video content. Specifically, the generation instruction includes information indicating the content expressed by the video content, such as "a man jogging", and subjects appearing in the video content, their actions, scenes, etc. The generation instruction may include information specifying the playback time of the video content to be generated, and may include information indicating a standard aspect ratio for the video content (hereinafter, sometimes referred to as a "standard aspect ratio").

[0022] The information processing device 1 acquires metadata ((2) in FIG. 1). The metadata is data describing a material. As an example, the metadata includes the aspect ratio of the material. The metadata may include a description describing the material, such as a subject captured in the material or a theme expressed by the material.

[0023] The information processing device 1 identifies the modification content of the material based on the generation instruction and the metadata ((3) in FIG. 1). As an example, the information processing device 1 compares the aspect ratio of the video content included in the generation instruction with the aspect ratio included in the metadata of the material, and identifies changing the aspect ratio of the material as the modification content of the material when the aspect ratio of the video content included in the generation instruction differs from the aspect ratio included in the metadata of the material. For example, when the aspect ratio of the material is "4:3" and the standard aspect ratio of the video content is "16:9", the information processing device 1 identifies "changing the aspect ratio to 16:9" as the modification content.

[0024] The information processing device 1 generates a prompt for correcting the material based on the identified correction content ((4) in FIG. 1). The prompt is an instruction written in natural language and is an instruction to be given to the machine learning model. For example, if the identified correction content is a change in aspect ratio and the reference aspect ratio included in the generation instruction is "16:9", a prompt is generated stating "Please change the aspect ratio of the input content to 16:9".

[0025] The information processing device 1 transmits the generated prompt to the generation device 3 ((5) in FIG. 1). The information processing device 1 may display the generated prompt on the information terminal 2 and prompt the user for confirmation. When the user performs a confirmation operation on the information terminal 2, the information processing device 1 may transmit the generated prompt and the accepted material to the generation device 3 and cause the material to be modified. As an example, the generation device 3 modifies the material based on the prompt and causes the information terminal 2 to display the modified material.

[0026] By configuring the information processing system S in this way, it is possible to reduce the burden of modifying materials to suit the content.

[0027] [Configuration of information processing device 1] 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has a reception unit 131, an acquisition unit 132, an identification unit 133, a generation unit 134, and a determination unit 135.

[0028] The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), a hard disk drive, etc. The storage unit 12 stores in advance a program to be executed by the control unit 13.

[0029] The control unit 13 is a processor such as a CPU (Central Processing Unit). The control unit 13 executes a program stored in the storage unit 12 to function as a reception unit 131, an acquisition unit 132, a specification unit 133, a generation unit 134, and a determination unit 135.

[0030] The reception unit 131 receives materials for constituting the video content and a generation instruction for generating the video content. The reception unit 131 displays a reception screen shown in FIG. 3 on the information terminal 2. The reception screen is a screen for receiving a generation instruction. The reception screen includes, as an example, an interface O1 for selecting a material, an interface O2 for specifying the aspect ratio of the generated video, and an interface O3 for specifying the content of the video. When the user presses the send button O4, the reception unit 131 acquires a generation instruction including the content specified by the user. Note that the reception screen may include an interface for selecting a template associated with video settings such as the aspect ratio and the playback time of the video, instead of the interface for selecting the aspect ratio.

[0031] The acquisition unit 132 acquires metadata about the material. The acquisition unit 132 may acquire the metadata about the material from the information terminal 2. The acquisition unit 132 may acquire the metadata from a material management server (not shown) that manages the material. In this case, the acquisition unit 132 accepts the material ID of the specified material from the information terminal 2, and acquires the metadata corresponding to the material ID accepted by the acceptance unit 131 from the material management server. The determination unit 135 determines whether or not modification is necessary to apply the material to the video content based on the metadata and the generation instruction. The determination unit 135 compares the metadata with the generation instruction to determine whether or not modification is necessary. The determination unit 135 may determine whether or not modification is necessary by comparing the aspect ratios included in the metadata and the generation instruction, or may determine whether or not modification is necessary by comparing the contents of the material and the video content included in the metadata and the generation instruction, respectively. The identification unit 133 identifies the content of the modification when the determination unit 135 determines that modification of the material is necessary.

[0032] The determination unit 133 determines the modification content for applying the material to the video content based on the generation instruction and the metadata of the material. As an example, when the aspect ratio indicated by the metadata differs from the reference aspect ratio, the determination unit 133 determines the change of the aspect ratio of the material as the modification content.

[0033] The generation unit 134 generates a prompt for inputting an instruction to correct the material to the image generation AI based on the identified correction content. The generation unit 134 generates a prompt for changing the aspect ratio of the material to the aspect ratio identified by the identification unit 133. As an example, the storage unit 12 stores a template for generating a prompt. The generation unit 134 generates a prompt by fitting the identified correction content to the template stored in the storage unit 12. In addition, the storage unit 12 stores a trained model (hereinafter referred to as a "prompt generation model") that has been trained to output a prompt written in a natural language when the correction content is input, and the generation unit 134 inputs the correction content identified by the identification unit 133 into the prompt generation model and causes it to output a prompt corresponding to the correction content.

[0034] When the specifying unit 133 specifies a change in aspect ratio as the modification content, the generating unit 134 generates a prompt based on the generating instruction to instruct the user to supplement a missing portion for changing the aspect ratio of the material based on the modification content based on a description explaining the video content indicated by the generating instruction. As an example, when generating a vertical video content based on a horizontal image material as shown in FIG. 4, a region R to which the material cannot be applied is generated in the video content. As an example, the specifying unit 133 may specify as the region R an area that cannot be filled with the material when the material is placed in the center. In this case, the specifying unit 133 specifies that the specified region R is to be supplemented based on the content of the video content included in the generating instruction as the modification content. In this case, the generating unit 134 generates a prompt that instructs the user to generate the region R specified by the specifying unit 133 based on the content of the video included in the generating instruction. In this case, as an example, the generating unit 134 generates a prompt that states, "Please change the aspect ratio of the input content to 9:16. Please generate the portion not filled by the input content based on 'a man running'."

[0035] In this way, it is desirable that the content to be supplemented in the region R is natural in relation to the material. Therefore, the prompt generation model may learn the user's evaluation of the generated prompt.

[0036] More specifically, the information processing device 1 acquires an evaluation indicating whether a user who has viewed a material modified by an image generation AI based on a prompt generated by a prompt generation model feels the modified material natural. The user's evaluation may be expressed as a binary value indicating whether it feels natural or not, or may be expressed as information indicating the degree to which it feels natural. Note that the user's evaluation is not limited to these examples. As an example, the generation model learns training data that associates the prompt generated by the prompt generation model with the evaluation. In addition, the prompt generation model may learn training data that includes prompts that are evaluated by users at or above a predetermined threshold.

[0037] When the identification unit 133 identifies a change in aspect ratio as the modification content, the generation unit 134 generates a prompt based on the generation instruction to instruct cutting out a portion that will become surplus when the aspect ratio of the material is changed based on the modification content. In this case, the identification unit 133 identifies an area of ​​the material that will become surplus by changing the aspect ratio, and generates a prompt to instruct trimming the surplus area.

[0038] If the material is a video and the playback time of the material is different from the playback time of the video content associated with the template, a prompt may be generated that indicates how to handle the missing or surplus time. In this case, as an example, the template is associated with a method of handling a case where the playback time of the material is shorter or longer than the playback time of the video content. As an example, when the playback time of the material is shorter than the playback time of the video content, it is associated with playing the material in a loop or generating a digest video in which the content indicated by the material is edited to fill the missing time. Also, when the playback time of the material is longer than the playback time of the video content, it may be associated with deleting a predetermined part of the material (e.g., the end). When the playback time of the material differs from the playback time of the video content associated with the template, the identification unit 133 refers to a method of dealing with a shortage or excess time associated with the template and identifies the correction content.

[0039] The generating unit 134 generates a prompt based on the correction content identified by the identifying unit 133. As an example, when the playback time of the material is 9 minutes and the playback time of the video content is 10 minutes, and the method of handling the surplus time is to generate a digest video, the generating unit 134 generates a prompt stating "Please generate a digest that summarizes the input content in one minute, and connect the generated digest after the input content to generate a video." In addition, when the playback time of the material is 10 minutes and the playback time of the video content is 60 minutes, and the method of handling the surplus time is to play in a loop, the generating unit 134 generates a prompt stating "Please generate a video that repeats the input content six times."

[0040] The information processing device 1 may be configured to generate prompts to correct content captured in material that is a photographic image.

[0041] When the description describing the subject of the photographic image indicated by the metadata does not match the description describing the video content indicated by the generation instruction, the generation unit 134 generates a prompt to modify the photographic image based on the description describing the video content indicated by the generation instruction. As an example, when the metadata indicates that the content included in the material image is "5 dumplings" and the generation instruction indicates that the content of the video content is "6 dumplings", the generation unit 134 generates a prompt to modify the material to 6 dumplings.

[0042] The information processing device 1 may be configured to generate a prompt that further includes, as an instruction, a restriction on generating content.

[0043] The generation instruction further includes restrictions indicating matters that limit the expression in the video content. For example, the restrictions are to prevent a brand logo image, a character that is the subject of copyright, and the like from being included in the generated content. For example, the restrictions may be to prevent contents that suggest violence, crime, and the like from being included in the content. As an example, an interface for selecting restrictions to be specified for the generated content may be arranged on the reception screen, and the reception unit 131 may receive the restrictions selected by the user on the reception screen as the generation instruction.

[0044] The generator 134 generates a prompt that further includes a restriction. As an example, the generator 134 may generate a prompt that includes a sentence specifying the content and a sentence indicating the restriction.

[0045] [Processing flow in information processing device 1] 5 is a flowchart showing a flow of processing in the information processing device 1. The reception unit 131 receives a material and a generation instruction (S01). The acquisition unit 132 acquires metadata about the material received by the reception unit 131 (S02).

[0046] The identification unit 133 identifies the correction content of the material based on the generation instruction and the metadata of the material (S03). The generation unit 134 generates a prompt for inputting an instruction to correct the material to the image generation AI based on the identified correction content (S04). Then, the information processing device 1 ends the process.

[0047] [Effects of information processing device 1] By configuring the information processing device 1 as described above, it is possible to reduce the burden of modifying materials so that they match the content.

[0048] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote industry, innovation and infrastructure."

[0049] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination combines the effect of the original embodiment. [Explanation of symbols]

[0050] 1. Information processing device 2. Information terminal 3 Generator 11 Communications Department 12 Storage section 13 Control section 131 Reception 132 Acquisition Department 133 Specific part 134 Generation Department 135 Judgment Department

Claims

1. a receiving unit that receives a source of photographic images for constituting video content and a generation instruction for generating the video content, the generation instruction including an explanation of the content of the video content; an acquisition unit that acquires metadata, which is data describing the material, the metadata including a description of the subject of the photographic image; a specification unit that specifies a modification content for applying the material to the video content based on the generation instruction and metadata of the material; A generation unit that generates a prompt for inputting an instruction to modify the material to an image generation AI based on the identified modification content; having When a description of a subject of the photographic image indicated by the metadata does not match a description of the video content indicated by the generation instruction, the generation unit generates a prompt for modifying the material, which is the photographic image, based on the description of the video content indicated by the generation instruction. Information processing device.

2. the metadata includes an aspect ratio of the material; the generation instruction includes a reference aspect ratio that is a reference aspect ratio for the video content, the identifying unit, when the aspect ratio indicated by the metadata is different from the reference aspect ratio, identifies a change in the aspect ratio of the material as the modification content; the generating unit generates a prompt for changing the aspect ratio of the material to the aspect ratio specified by the specifying unit; The information processing device according to claim 1 .

3. The generation instruction includes a description explaining the video content, When the specification unit specifies a change in aspect ratio as the modification content, the generation unit generates, based on the generation instruction, a prompt for instructing to supplement a portion that is insufficient for changing the aspect ratio of the material based on the modification content, based on a description explaining the video content indicated by the generation instruction. The information processing device according to claim 2 .

4. When the specification unit specifies a change in aspect ratio as the modification content, the generation unit generates, based on the generation instruction, a prompt for instructing to cut off a portion that would become redundant if the aspect ratio of the material is changed based on the modification content. The information processing device according to claim 2 .

5. a determination unit that determines whether or not the material needs to be modified to apply to the video content based on the metadata and the generation instruction, The identification unit identifies the content of the correction when the determination unit determines that the material needs to be corrected. The information processing device according to claim 1 .

6. The generation instruction further includes a restriction indicating a restriction on an expression of the video content, The generating unit generates a prompt further including the restriction. The information processing device according to claim 1 .

7. The computer executes receiving a source of photographic images for constituting video content and a generation instruction for generating the video content, the generation instruction including a description of the content of the video content; obtaining metadata, which is descriptive data about the material, the metadata including a description of the content of a subject of the photographic image; determining, based on the generation instructions and metadata of the material, modifications to be applied to the material to the video content; generating a prompt for inputting instructions to the image generation AI to modify the material based on the identified modification content; having In the generating step, if a description of the subject of the photographic image indicated by the metadata does not match a description of the content of the video content indicated by the generation instruction, a prompt is generated for modifying the material, which is the photographic image, based on the description of the content of the video content indicated by the generation instruction. Information processing methods.

8. On the computer, receiving a source of photographic images for constituting video content and a generation instruction for generating the video content, the generation instruction including a description of the content of the video content; obtaining metadata, which is descriptive data about the material, the metadata including a description of the content of a subject of the photographic image; determining, based on the generation instructions and metadata of the material, modifications to be applied to the material to the video content; generating a prompt for inputting instructions to the image generation AI to modify the material based on the identified modification content; having In the generating step, if a description of the subject of the photographic image indicated by the metadata does not match a description of the content of the video content indicated by the generation instruction, a prompt is generated for modifying the material, which is the photographic image, based on the description of the content of the video content indicated by the generation instruction. A program that executes the following.

Images