Information processing device, information processing method and program

Abstract

To reduce a burden of modifying materials to fit content.SOLUTION: An information processing device 1 includes: a receiving unit 131 that receives materials for constructing video content and generation instructions for generating the video content; an acquiring unit 132 that acquires metadata, the data describing the materials; a specifying unit 133 that specifies a modification content for applying the materials to the video content on the basis of the generation instructions and the metadata of the materials; and a generating unit 134 that generates prompts for inputting an instruction to modify the materials into an image generation AI on the basis of the specified modification content.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art There is known a device that composes content by combining materials (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2024-027453 Summary of the Invention [Problem to be solved by the invention]

[0004] Such materials may not be suitable for the desired video content, which limits the creation of content and creates the problem of a heavy burden of manually correcting the materials.

[0005] The present invention has been made in consideration of these points, and aims to reduce the burden of modifying materials to fit the content. [Means for solving the problem]

[0006] An information processing device according to a first aspect of the present invention includes a receiving unit that receives materials for constituting video content and a generation instruction for generating the video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content and a description explaining what the video content expresses; an acquiring unit that acquires metadata that is data describing the materials, the metadata including the aspect ratio of the materials; an identifying unit that identifies correction content for applying the materials to the video content based on the generation instruction and the metadata of the materials; and and a generation unit that generates a prompt for inputting instructions to modify the material to the image generation AI based on the modification content, wherein the identification unit identifies a change in the aspect ratio of the material as the modification content when the aspect ratio indicated by the metadata differs from the reference aspect ratio, and when the identification unit identifies a change in aspect ratio as the modification content, the generation unit generates a prompt based on the generation instruction to instruct the image generation AI to supplement the missing part that is required to change the aspect ratio of the material based on the modification content based on a description that explains the content expressed by the video content indicated by the generation instruction.

[0007] When the identification unit identifies a change in aspect ratio as the modification content, the generation unit may generate a prompt based on the generation instruction to instruct cutting out any portion that would become excess if the aspect ratio of the material were changed based on the modification content.

[0008] The material includes a photographic image, the metadata includes a description describing the subject of the photographic image, and the generation unit may generate a prompt to modify the photographic image based on the description describing the content expressed by the video content indicated by the generation instruction when the description describing the subject of the photographic image indicated by the metadata does not match the description describing the content expressed by the video content indicated by the generation instruction.

[0009] The device may further include a judgment unit that judges whether or not modification is necessary to apply the material to the video content based on the metadata and the generation instructions, and the identification unit may identify the content of the modification when the judgment unit determines that modification of the material is necessary.

[0010] The generation instruction may further include a restriction indicating restrictions on expressions in the video content, and the generation unit may generate a prompt further including the restriction.

[0011] The generation unit generates a prompt by inputting the correction content identified by the identification unit into a prompt generation model that has been trained to output a prompt for inputting instructions to the image generation AI to modify the material when correction content for applying the material to video content is input, and the prompt generation model may be configured to learn training data that associates the learning prompt with an evaluation of the image generated by the image generation model based on the learning prompt.

[0012] In a second aspect of the present invention, an information processing method includes the steps of: receiving materials for constituting video content; and a generation instruction for generating the video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content and a description explaining what the video content expresses; acquiring metadata that is data describing the materials, the metadata including the aspect ratio of the materials; specifying corrections for applying the materials to the video content based on the generation instruction and the metadata of the materials; and modifying the video content based on the specified corrections. and generating a prompt for inputting instructions to modify the material to the image generation AI based on the aspect ratio indicated by the metadata. In the specifying step, if the aspect ratio indicated by the metadata differs from the reference aspect ratio, a change in the aspect ratio of the material is specified as the modification content. In the generating step, if a change in aspect ratio is specified as the modification content in the specifying step, a prompt is generated to instruct the AI ​​to supplement, based on the generation instructions, the missing part for modifying the aspect ratio of the material based on the modification content, based on a description explaining the content expressed by the video content indicated by the generation instructions.

[0013] In a third aspect of the present invention, a program includes the steps of: receiving, in a computer, materials for constituting video content; and a generation instruction for generating the video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content and a description explaining what the video content expresses; acquiring metadata that is data describing the materials, the metadata including the aspect ratio of the materials; specifying corrections for applying the materials to the video content based on the generation instruction and the metadata of the materials; and, based on the specified corrections, performing the above-mentioned processing. and generating a prompt for inputting instructions to modify the material into the image generation AI, wherein in the specifying step, if the aspect ratio indicated by the metadata differs from the reference aspect ratio, a change in the aspect ratio of the material is specified as the modification content, and in the generating step, if a change in aspect ratio is specified as the modification content in the specifying step, a prompt is generated to instruct the AI ​​to supplement, based on the generation instructions, the missing part required to change the aspect ratio of the material based on the modification content, based on a description explaining the content expressed by the video content indicated by the generation instructions.

[0014] [Effects of the Invention]

[0015] The present invention has the effect of reducing the burden of modifying materials to fit content. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram for explaining an overview of an information processing device 1 according to an embodiment. [Figure 2] 1 is a block diagram showing a configuration of an information processing device 1. FIG. [Figure 3] FIG. 10 is a diagram showing an example of a reception screen displayed by a reception unit 131. [Figure 4]FIG. 10 is a diagram illustrating an example of processing by an identification unit 133. [Figure 5] 3 is a flowchart showing the flow of processing in the information processing device 1. DETAILED DESCRIPTION OF THE INVENTION

[0017] [Outline of Information Processing System S] FIG. 1 is a diagram illustrating an overview of an information processing device 1 according to an embodiment. The information processing system S is a system for providing video content. The information processing system S generates video content based on materials specified by a user. The information processing system S includes an information processing device 1, an information terminal 2, and a generation device 3. The materials are content elements that make up the video content. The materials are video data, still image data, text data, audio data, music data, etc.

[0018] The information processing device 1 is a device for generating a prompt for giving instructions to a machine learning model. The information processing device 1 determines whether or not the material needs to be modified based on the content specified by the user and the metadata of the material specified by the user. If modification is necessary, the information processing device 1 generates a prompt for instructing the generation device 3 to modify the material.

[0019] The information terminal 2 is a terminal used by a user of the information processing system S. The information terminal 2 is, for example, a smartphone, a tablet, or a personal computer. The information terminal 2 transmits instructions to the information processing device 1 regarding the details of the content to be generated.

[0020] The generation device 3 generates video content, still images, etc. based on the received instructions. The generation device 3 has a generative model (hereinafter sometimes referred to as "image generation AI") that is a trained model trained to generate video or still images based on instructions written in natural language, and outputs video or still images generated by the generative model based on the input instructions.

[0021] The processing of the information processing device 1 will be described. The information processing device 1 acquires materials and a generation instruction from the information terminal 2 ((1) in FIG. 1). The generation instruction is an instruction to generate video content. As an example, the generation instruction includes a description that explains the video content. Specifically, the generation instruction includes information indicating the content expressed by the video content, such as "a man jogging," as well as subjects appearing in the video content, their actions, and the scene. The generation instruction may include information specifying the playback time of the video content to be generated, and may include information indicating the standard aspect ratio for the video content (hereinafter sometimes referred to as the "standard aspect ratio").

[0022] The information processing device 1 acquires metadata ((2) in FIG. 1). The metadata is data describing the material. For example, the metadata includes the aspect ratio of the material. The metadata may also include descriptions describing the material, such as subjects captured in the material or the theme expressed by the material.

[0023] The information processing device 1 identifies the modification content of the material based on the generation instruction and the metadata ((3) in FIG. 1). As an example, the information processing device 1 compares the aspect ratio of the video content included in the generation instruction with the aspect ratio included in the metadata of the material, and if the aspect ratio of the video content included in the generation instruction differs from the aspect ratio included in the metadata of the material, identifies changing the aspect ratio of the material as the modification content of the material. For example, if the aspect ratio of the material is "4:3" and the standard aspect ratio of the video content is "16:9," the information processing device 1 identifies "changing the aspect ratio to 16:9" as the modification content.

[0024] The information processing device 1 generates a prompt for correcting the material based on the identified correction content ((4) in FIG. 1). The prompt is an instruction written in natural language and is an instruction to be given to the machine learning model. For example, if the identified correction content is a change in aspect ratio and the reference aspect ratio included in the generation instruction is "16:9," the information processing device 1 generates a prompt stating "Please change the aspect ratio of the input content to 16:9."

[0025] The information processing device 1 transmits the generated prompt to the generation device 3 ((5) in FIG. 1). The information processing device 1 may display the generated prompt on the information terminal 2 to prompt the user for confirmation. When the user performs a confirmation operation on the information terminal 2, the information processing device 1 may transmit the generated prompt and the accepted material to the generation device 3 to modify the material. As an example, the generation device 3 modifies the material based on the prompt and displays the modified material on the information terminal 2.

[0026] By configuring the information processing system S in this way, it is possible to reduce the burden of modifying materials to suit the content.

[0027] [Configuration of information processing device 1] 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has a reception unit 131, an acquisition unit 132, an identification unit 133, a generation unit 134, and a determination unit 135.

[0028] The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), a hard disk drive, etc. The storage unit 12 stores in advance a program to be executed by the control unit 13.

[0029] The control unit 13 is a processor such as a CPU (Central Processing Unit), etc. The control unit 13 executes a program stored in the storage unit 12, thereby functioning as a reception unit 131, an acquisition unit 132, an identification unit 133, a generation unit 134, and a determination unit 135.

[0030] The reception unit 131 receives materials for constructing video content and a generation instruction for generating video content. The reception unit 131 displays the reception screen shown in FIG. 3 on the information terminal 2. The reception screen is a screen for receiving a generation instruction. The reception screen includes, as an example, an interface O1 for selecting materials, an interface O2 for specifying the aspect ratio of the generated video, and an interface O3 for specifying the content of the video. When the user presses the send button O4, the reception unit 131 acquires a generation instruction including the content specified by the user. Note that the reception screen may include an interface for selecting a template associated with video settings such as the aspect ratio and video playback time, instead of the interface for selecting the aspect ratio.

[0031] The acquisition unit 132 acquires metadata about the material. The acquisition unit 132 may acquire the metadata about the material from the information terminal 2. The acquisition unit 132 may acquire the metadata from a material management server (not shown) that manages the material. In this case, the acquisition unit 132 accepts the material ID of the specified material from the information terminal 2, and acquires the metadata corresponding to the material ID accepted by the acceptance unit 131 from the material management server. The determination unit 135 determines whether or not modification is necessary to apply the material to the video content based on the metadata and the generation instruction. The determination unit 135 compares the metadata with the generation instruction to determine whether or not modification is necessary. The determination unit 135 may determine whether or not modification is necessary by comparing the aspect ratios included in the metadata and the generation instruction, or may determine whether or not modification is necessary by comparing the content of the material and the video content included in the metadata and the generation instruction, respectively. The identification unit 133 identifies the content of the modification when the determination unit 135 determines that modification of the material is necessary.

[0032] The determination unit 133 determines the modification content for applying the material to the video content based on the generation instruction and the metadata of the material. As an example, when the aspect ratio indicated by the metadata differs from the reference aspect ratio, the determination unit 133 determines that the modification content is a change in the aspect ratio of the material.

[0033] The generation unit 134 generates a prompt for inputting an instruction to modify the material to the image generation AI based on the specified modification content. The generation unit 134 generates a prompt for changing the aspect ratio of the material to the aspect ratio specified by the specification unit 133. As an example, the storage unit 12 stores a template for generating a prompt. The generation unit 134 generates a prompt by fitting the specified modification content into the template stored in the storage unit 12. The storage unit 12 also stores a trained model (hereinafter referred to as a "prompt generation model") that has been trained to output a prompt written in natural language when the modification content is input, and the generation unit 134 inputs the modification content specified by the specification unit 133 into the prompt generation model and causes it to output a prompt corresponding to the modification content.

[0034] When the identification unit 133 identifies a change in aspect ratio as the modification content, the generation unit 134 generates a prompt based on the generation instruction to instruct the user to fill in the missing portion of the content due to the aspect ratio of the material being changed based on the modification content, based on a description describing the video content indicated by the generation instruction. For example, when generating portrait-oriented video content based on a landscape-oriented image material as shown in FIG. 4, a region R in the video content is generated to which the material cannot be applied. For example, the identification unit 133 may identify as region R an area that cannot be filled with the material when the material is positioned in the center. In this case, the identification unit 133 identifies, as the modification content, filling in the identified region R based on the content of the video content included in the generation instruction. In this case, the generation unit 134 generates a prompt instructing the user to generate the identified region R based on the content of the video content included in the generation instruction. In this case, for example, the generation unit 134 generates a prompt stating, "Please change the aspect ratio of the input content to 9:16. Please generate the portion not filled by the input content based on 'a running man.'"

[0035] In this way, it is desirable that the content to be added to region R be natural in relation to the material. Therefore, the prompt generation model may learn the user's evaluation of the generated prompt.

[0036] More specifically, the information processing device 1 acquires an evaluation indicating whether a user who has viewed material modified by an image generation AI based on a prompt generated by a prompt generation model feels the modified material is natural. The user's evaluation may be expressed as a binary value indicating whether the material is perceived as natural or not, or may be expressed as information indicating the degree to which the material is perceived as natural. Note that the user's evaluation is not limited to these examples. As an example, the generation model learns training data that associates the prompt generated by the prompt generation model with the evaluation. Furthermore, the prompt generation model may learn training data that includes prompts that result in user evaluations above a predetermined threshold.

[0037] When the identification unit 133 identifies a change in aspect ratio as the modification content, the generation unit 134 generates a prompt based on the generation instruction to instruct trimming of a portion that would become surplus if the aspect ratio of the material were changed based on the modification content. In this case, the identification unit 133 identifies an area of ​​the material that would become surplus by changing the aspect ratio, and generates a prompt to instruct trimming of the surplus area.

[0038] If the material is a video and the playback time of the material differs from the playback time of the video content associated with the template, a prompt may be generated indicating how to handle the missing or excess time. In this case, for example, the template may be associated with a method for handling cases where the playback time of the material is shorter or longer than the playback time of the video content. For example, if the playback time of the material is shorter than the playback time of the video content, it may be associated with looping the material or generating a digest video by editing the content of the material to fill the missing time. Furthermore, if the playback time of the material is longer than the playback time of the video content, it may be associated with deleting a predetermined portion of the material (e.g., the end). When the playback time of the material differs from the playback time of the video content associated with the template, the identification unit 133 refers to the method of handling the insufficient or excess time associated with the template and identifies the correction content.

[0039] The generation unit 134 generates a prompt based on the correction content identified by the identification unit 133. For example, if the playback time of the material is 9 minutes and the playback time of the video content is 10 minutes, and the method for handling the excess time is to generate a digest video, the generation unit 134 generates a prompt stating, "Please generate a digest that summarizes the input content into one minute, and connect the generated digest after the input content to generate a video." Alternatively, if the playback time of the material is 10 minutes and the playback time of the video content is 60 minutes, and the method for handling the excess time is to play the input content in a loop, the generation unit 134 generates a prompt stating, "Please generate a video that repeats the input content six times."

[0040] The information processing device 1 may be configured to generate a prompt to correct content that appears in material that is a photographic image.

[0041] When the description describing the subject of the photographic image indicated by the metadata does not match the description describing the video content indicated by the generation instruction, the generation unit 134 generates a prompt to correct the photographic image based on the description describing the video content indicated by the generation instruction. As an example, when the metadata indicates that the content included in the material image is "5 dumplings" and the generation instruction indicates that the content of the video content is "6 dumplings," the generation unit 134 generates a prompt to correct the material to 6 dumplings.

[0042] The information processing device 1 may be configured to generate a prompt that further includes, as an instruction, a restriction on content generation.

[0043] The generation instruction further includes restrictions indicating restrictions on expressions in the video content. For example, the restrictions may be to prevent brand logo images, copyrighted characters, etc. from being included in the generated content. For example, the restrictions may be to prevent content suggesting violence, crime, etc. from being included in the content. As an example, the reception screen may include an interface for selecting restrictions to be specified for the content to be generated, and the reception unit 131 may receive the restrictions selected by the user on the reception screen as the generation instruction.

[0044] The generator 134 generates a prompt that further includes a restriction. For example, the generator 134 may generate a prompt that includes a sentence specifying the content followed by a sentence indicating the restriction.

[0045] [Processing flow in information processing device 1] 5 is a flowchart showing the flow of processing in the information processing device 1. The receiving unit 131 receives a material and a generation instruction (S01). The acquiring unit 132 acquires metadata about the material received by the receiving unit 131 (S02).

[0046] The identification unit 133 identifies the content of the material to be corrected based on the generation instruction and the metadata of the material (S03). The generation unit 134 generates a prompt for inputting an instruction to correct the material to the image generation AI based on the identified content of the correction (S04). The information processing device 1 then ends the process.

[0047] [Effects of information processing device 1] By configuring the information processing device 1 as described above, it is possible to reduce the burden of modifying materials to suit the content.

[0048] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0049] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]

[0050] 1. Information processing equipment 2. Information terminal 3 Generator 11 Communications Department 12 Storage section 13 Control Unit 131 Reception 132 Acquisition Department 133 Specific part 134 Generation Department 135 Judgment Department

Claims

1. a receiving unit that receives materials for constituting video content and a generation instruction for generating the video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content and a description explaining the content expressed by the video content; an acquisition unit that acquires metadata that is data describing the material, the metadata including an aspect ratio of the material; an identification unit that identifies a modification to be applied to the material to the video content based on the generation instruction and metadata of the material; A generation unit that generates a prompt for inputting an instruction to modify the material to the image generation AI based on the specified modification content; and the specifying unit specifies a change in the aspect ratio of the material as the modification content when the aspect ratio indicated by the metadata and the reference aspect ratio are different; When the specification unit specifies a change in aspect ratio as the modification content, the generation unit generates, based on the generation instruction, a prompt for instructing to supplement, based on a description explaining the content expressed by the video content indicated by the generation instruction, a portion that is missing due to the aspect ratio of the material being changed based on the modification content. Information processing device.

2. When the specification unit specifies a change in aspect ratio as the modification content, the generation unit generates, based on the generation instruction, a prompt for instructing to cut off a portion that will become redundant when the aspect ratio of the material is changed based on the modification content. The information processing device according to claim 1 .

3. the material includes photographic images; the metadata includes descriptions describing the subject of the photographic image; When a description describing a subject of the photographic image indicated by the metadata does not match a description describing a content expressed by the video content indicated by the generation instruction, the generation unit generates a prompt to modify the photographic image based on the description describing a content expressed by the video content indicated by the generation instruction. The information processing device according to claim 1 .

4. a determination unit that determines whether or not the material needs to be modified to apply it to the video content, based on the metadata and the generation instruction; the specifying unit specifies the content of the correction when the determining unit determines that the material needs to be corrected; The information processing device according to claim 1 .

5. the generation instruction further includes a restriction indicating a restriction on an expression in the video content; the generating unit generates a prompt further including the restriction. The information processing device according to claim 1 .

6. The generation unit generates a prompt by inputting the correction content identified by the identification unit into a prompt generation model that has learned to output a prompt for inputting an instruction to correct the material to the image generation AI when correction content for applying the material to video content is input; the prompt generation model learns training data that associates training prompts with evaluations of images generated by an image generation model based on the training prompts. The information processing device according to claim 1 .

7. The computer executes receiving a generation instruction for generating video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content, and a description explaining the content expressed by the video content; acquiring metadata that is data describing the material, the metadata including an aspect ratio of the material; determining, based on the generation instruction and metadata of the material, modifications to be applied to the material to the video content; generating a prompt for inputting instructions to modify the material to the image generation AI based on the identified modification content; and In the step of specifying, if the aspect ratio indicated by the metadata is different from the reference aspect ratio, a change in the aspect ratio of the material is specified as the modification content; In the generating step, if a change in aspect ratio is identified as the modification content in the identifying step, a prompt is generated based on the generation instruction to instruct the user to supplement a missing part that is required to change the aspect ratio of the material based on the modification content, based on a description that explains the content expressed by the video content indicated by the generation instruction. Information processing methods.

8. On the computer, receiving a generation instruction for generating video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content, and a description explaining the content expressed by the video content; acquiring metadata that is data describing the material, the metadata including an aspect ratio of the material; determining, based on the generation instruction and metadata of the material, modifications to be applied to the material to the video content; generating a prompt for inputting instructions to modify the material to the image generation AI based on the identified modification content; Execute In the step of specifying, if the aspect ratio indicated by the metadata is different from the reference aspect ratio, a change in the aspect ratio of the material is specified as the modification content; In the generating step, if a change in aspect ratio is identified as the modification content in the identifying step, a prompt is generated based on the generation instruction to instruct the user to supplement a missing part that is required to change the aspect ratio of the material based on the modification content, based on a description that explains the content expressed by the video content indicated by the generation instruction. program.

Images