Information Processing Apparatus, Information Processing Method, and Program

Abstract

Reduce the burden of modifying materials to match the content. 【Solution means】 A receiving unit 131 that receives materials for composing video content and a generation instruction for generating video content, an acquisition unit 132 that acquires metadata, which is data described about the materials, a specifying unit 133 that specifies the correction content for applying the materials to the video content based on the generation instruction and the metadata of the materials, and a generation unit 134 that generates a prompt for inputting an instruction to correct the materials to the image generation AI based on the specified correction content. The information processing apparatus 1 has these components.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] An apparatus for constructing content by combining materials is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such a case, the materials may not be suitable for the desired video content, resulting in limitations in content generation, while at the same time, there is a problem that the burden of manually modifying the materials is large.

[0005] Therefore, the present invention has been made in view of these points, and an object thereof is to reduce the burden of modifying materials so as to be suitable for content.

Means for Solving the Problems

[0006] In the information processing apparatus according to the first aspect of the present invention, a receiving unit that receives a generation instruction for generating the video content, which includes materials for constructing the video content, a reference aspect ratio that is a reference aspect ratio in the video content, and a description that describes the content represented by the video content; an acquisition unit that acquires metadata that is data described about the material and includes the aspect ratio of the material; a specifying unit that specifies correction content for applying the material to the video content based on the generation instruction and the metadata of the material; and a generation unit that generates a prompt for inputting an instruction to correct the material to an image generation AI based on the specified correction content. When the aspect ratio indicated by the metadata is different from the reference aspect ratio, the specifying unit specifies the change in the aspect ratio of the material as the correction content. When the specifying unit specifies the change in the aspect ratio as the correction content, the generation unit generates a prompt for instructing to complement, based on the description that describes the content represented by the video content indicated by the generation instruction, a portion that is insufficient for changing the aspect ratio of the material based on the correction content according to the generation instruction.

[0007] When the specifying unit specifies the change in the aspect ratio as the correction content, the generation unit may generate a prompt for instructing to cut off a portion that becomes surplus when the aspect ratio of the material is changed based on the correction content according to the generation instruction.

[0008] The material includes a photographic image, the metadata includes a description that describes the subject of the photographic image, and when the description that describes the subject of the photographic image indicated by the metadata does not match the description that describes the content represented by the video content indicated by the generation instruction, the generation unit may generate a prompt for correcting the photographic image based on the description that describes the content represented by the video content indicated by the generation instruction.

[0009] The apparatus further includes a determination unit configured to determine whether a correction is necessary to apply the material to the video content based on the metadata and the generation instruction. The specifying unit may specify the correction content when the determination unit determines that the correction of the material is necessary.

[0010] The generation instruction further includes a restricted item indicating an item whose expression is restricted in the video content, and the generation unit may generate a prompt further including the restricted item.

[0011] The generation unit inputs the correction content specified by the specifying unit into a prompt generation model trained to output a prompt for inputting an instruction to correct the material to the image generation AI when a correction content for applying the material to the video content is input, and the prompt generation model may be configured to learn teacher data associating a learning prompt with an evaluation of an image generated by an image generation model based on the learning prompt.

[0012] In the information processing method according to the second aspect of the present invention, a material for constituting video content, which is executed by a computer, and a generation instruction for generating the video content, including a reference aspect ratio that is a reference aspect ratio in the video content, and a description explaining the content represented by the video content, are received; metadata that is data described about the material and includes the aspect ratio of the material is acquired; based on the generation instruction and the metadata of the material, a correction content for applying the material to the video content is specified; and a prompt for inputting an instruction to correct the material to an image generation AI is generated based on the specified correction content. In the specifying step, when the aspect ratio indicated by the metadata is different from the reference aspect ratio, the change in the aspect ratio of the material is specified as the correction content. In the generating step, when the change in the aspect ratio is specified as the correction content in the specifying step, a prompt for instructing to complement, based on the description explaining the content represented by the video content indicated by the generation instruction, a portion lacking for changing the aspect ratio of the material based on the correction content is generated based on the generation instruction.

[0013] In the program according to the third aspect of the present invention, the computer is caused to execute steps of receiving the generation instruction including materials for constituting video content, a generation instruction for generating the video content, a reference aspect ratio which is the aspect ratio serving as a reference in the video content, and a description explaining the content represented by the video content; obtaining metadata which is data described about the materials and includes the aspect ratio of the materials; specifying correction details for applying the materials to the video content based on the generation instruction and the metadata of the materials; and generating a prompt for inputting an instruction to correct the materials to an image generation AI based on the specified correction details. In the specifying step, when the aspect ratio indicated by the metadata is different from the reference aspect ratio, the change in the aspect ratio of the materials is specified as the correction details. In the generating step, when the change in the aspect ratio is specified as the correction details in the specifying step, a prompt is generated for instructing to complement, based on the description explaining the content represented by the video content indicated by the generation instruction, the portion lacking for changing the aspect ratio of the materials based on the correction details according to the generation instruction.

[0014]

Advantages of the Invention

[0015] According to the present invention, there is an effect of reducing the burden of correcting materials so as to conform to the content.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Mode for Carrying Out the Invention

[0017] [Overview of the Information Processing System S] FIG. 1 is a diagram for explaining the overview of the information processing apparatus 1 according to the embodiment. The information processing system S is a system for providing video content. The information processing system S generates video content based on the materials specified by the user. The information processing system S includes an information processing apparatus 1, an information terminal 2, and a generation apparatus 3. The material is content that becomes an element constituting the video content. The material is video data, still image data, text data, audio data, music data, or the like.

[0018] The information processing apparatus 1 is an apparatus for generating a prompt for giving an instruction to a machine learning model. The information processing apparatus 1 determines whether it is necessary to correct the material based on the content of the content specified by the user and the metadata of the material specified by the user. When correction is necessary, the information processing apparatus 1 generates a prompt for instructing the generation apparatus 3 to correct the material.

[0019] The information terminal 2 is a terminal used by the user of the information processing system S. The information terminal 2 is, for example, a smartphone, a tablet, or a personal computer. The information terminal 2 transmits an instruction regarding the content of the content generated by the information processing apparatus 1.

[0020] The generation apparatus 3 generates video content, still images, etc. based on the received instruction. The generation apparatus 3 has a generation model (hereinafter sometimes referred to as "image generation AI"), which is a learned model trained to generate a moving image or a still image based on an instruction described in natural language, and outputs the moving image or the still image generated by the generation model based on the input instruction.

[0021] The processing of the information processing apparatus 1 will be described. The information processing apparatus 1 acquires a material and a generation instruction from the information terminal 2 ((1) in FIG. 1). The generation instruction is an instruction to generate video content. As an example, the generation instruction includes a description explaining the video content. Specifically, in the generation instruction, information indicating the content expressed by the video content, such as "a man jogging", the subject appearing in the video content, its actions, scenes, etc. is included. The generation instruction may include information specifying the playback time of the video content to be generated, or may include information indicating the aspect ratio serving as a reference in the video content (hereinafter sometimes referred to as the "reference aspect ratio").

[0022] The information processing apparatus 1 acquires metadata ((2) in FIG. 1). The metadata is data described about the material. The metadata includes, as an example, the aspect ratio of the material. The metadata may include a description explaining the material indicating the subject captured in the material, the theme expressed by the material, etc.

[0023] The information processing apparatus 1 specifies the correction content of the material based on the generation instruction and the metadata ((3) in FIG. 1). As an example, the information processing apparatus 1 compares the aspect ratio of the video content included in the generation instruction with the aspect ratio included in the metadata of the material, and when the aspect ratio of the video content included in the generation instruction and the aspect ratio included in the metadata of the material are different, specifies changing the aspect ratio of the material as the correction content of the material. For example, when the aspect ratio of the material is "4:3" and the reference aspect ratio of the video content is "16:9", the information processing apparatus 1 specifies "changing the aspect ratio to 16:9" as the correction content.

[0024] The information processing device 1 generates a prompt for modifying materials based on the specified modification content ((4) in FIG. 1). The prompt is an instruction described in natural language and is an instruction given to the machine learning model. For example, when the specified modification content is a change in the aspect ratio and the reference aspect ratio included in the generation instruction is "16:9", a prompt described as "Please change the aspect ratio of the input content to 16:9" is generated.

[0025] The information processing device 1 transmits the generated prompt to the generation device 3 ((5) in FIG. 1). The information processing device 1 may display the generated prompt on the information terminal 2 to prompt the user for confirmation. When an operation for user confirmation is performed on the information terminal 2, the information processing device 1 may transmit the generated prompt and the received material to the generation device 3 to cause the material to be modified. As an example, the generation device 3 modifies the material based on the prompt and displays the modified material on the information terminal 2.

[0026] By configuring the information processing system S in this way, there is an effect of reducing the burden of modifying materials to conform to the content.

[0027] [Configuration of Information Processing Device 1] FIG. 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 includes a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 includes a reception unit 131, an acquisition unit 132, a specification unit 133, a generation unit 134, and a determination unit 135.

[0028] The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), a hard disk drive, etc. The storage unit 12 stores in advance a program executed by the control unit 13.

[0029] The control unit 13 is a processor such as a CPU (Central Processing Unit). By executing the program stored in the storage unit 12, the control unit 13 functions as a reception unit 131, an acquisition unit 132, a specification unit 133, a generation unit 134, and a determination unit 135.

[0030] The reception unit 131 receives materials for constructing video content and a generation instruction for generating video content. The reception unit 131 causes the information terminal 2 to display a reception screen shown in FIG. 3. The reception screen is a screen for receiving a generation instruction. In the reception screen, as an example, it includes an interface O1 for selecting materials, an interface O2 for specifying the aspect ratio of the generated video, and an interface O3 for specifying the content of the video. When the user presses the transmission button O4, the reception unit 131 acquires a generation instruction including the content specified by the user. Note that in the reception screen, instead of an interface for selecting an aspect ratio, an interface for selecting a template associated with settings of the video such as an aspect ratio and a playback time of the video may be provided.

[0031] The acquisition unit 132 acquires metadata about the materials. The acquisition unit 132 may acquire metadata about the materials from the information terminal 2. The acquisition unit 132 may acquire metadata from a material management server (not shown) that manages the materials. In this case, the acquisition unit 132 receives the material ID of the specified material from the information terminal 2 and acquires the metadata corresponding to the material ID received by the reception unit 131 from the material management server. The determination unit 135 determines whether it is necessary to correct the material for applying it to the video content based on the metadata and the generation instruction. The determination unit 135 compares the metadata and the generation instruction to determine whether correction is necessary. The determination unit 135 may compare the aspect ratios included in the metadata and the generation instruction to determine whether correction is required, or may compare the content of the materials and the video content included in the metadata and the generation instruction respectively to determine whether correction is required. The specification unit 133 specifies the correction content when the determination unit 135 determines that correction of the material is necessary.

[0032] Based on the generation instruction and the metadata of the material, the specific part 133 identifies the modification content for applying the material to the video content. As an example, when the aspect ratio indicated by the metadata is different from the reference aspect ratio, the specific part 133 identifies the change in the aspect ratio of the material as the modification content.

[0033] Based on the identified modification content, the generation part 134 generates a prompt for inputting an instruction to modify the material into the image generation AI. The generation part 134 generates a prompt for changing the aspect ratio of the material to the aspect ratio identified by the specific part 133. As an example, the storage part 12 stores a template for generating a prompt. The generation part 134 generates a prompt by fitting the identified modification content to the template stored in the storage part 12. Also, the storage part 12 stores a learned model (hereinafter referred to as "prompt generation model") that has been learned to output a prompt described in natural language when the modification content is input, and the generation part 134 inputs the modification content identified by the specific part 133 into the prompt generation model and causes the prompt corresponding to the modification content to be output.

[0034] When the specifying unit 133 specifies the change in the aspect ratio as the correction content, the generation unit 134 generates a prompt for instructing to complement, based on the generation instruction, the portion that is insufficient to change the aspect ratio of the material based on the correction content, based on the description explaining the video content indicated by the generation instruction. As an example, as shown in FIG. 4, when generating a vertical video content based on a horizontal image material, an area R where the material cannot be applied occurs in the video content. As an example, the specifying unit 133 may specify, as the area R, an area that cannot be filled by the material when the material is arranged in the center. In this case, the specifying unit 133 specifies, as the correction content, to complement the specified area R based on the content of the video content included in the generation instruction. In this case, the generation unit 134 generates a prompt for instructing to generate the area R specified by the specifying unit 133 based on the content of the video included in the generation instruction. In this case, as an example, the generation unit 134 generates a prompt described as "Please change the aspect ratio of the input content to 9:16. The portion not filled by the input content should be generated based on 'a running man'."

[0035] It is desirable that the content to complement the area R be natural in relation to the material in this way. Therefore, in the prompt generation model, the user's evaluation of the generated prompt may be learned.

[0036] More specifically, the information processing apparatus 1 acquires an evaluation indicating whether a user who views the material corrected by the image generation AI based on the prompt generated by the prompt generation model feels the corrected material is natural. The user's evaluation may be expressed in a binary value of whether it feels natural or may be expressed in information indicating the degree of feeling natural. Note that the user's evaluation is not limited to these examples. As an example, the generation model learns teacher data associating the prompt generated by the prompt generation model with the evaluation. Also, the prompt generation model may learn, as teacher data, a prompt for which the user's evaluation is equal to or higher than a predetermined threshold.

[0037] When the specifying unit 133 specifies the change in the aspect ratio as the content to be corrected, the generation unit 134 generates a prompt for instructing to cut off the surplus part when the aspect ratio of the material is changed based on the content to be corrected according to the generation instruction. In this case, the specifying unit 133 specifies the area of the material that becomes surplus by changing the aspect ratio, and generates a prompt for instructing to trim the surplus area.

[0038] When the material is a video and the playback time of the material is different from the playback time of the video content associated with the template, a prompt indicating how to process the insufficient or surplus time may be generated. In this case, as an example, in the template, the processing methods for the case where the playback time of the material is shorter or longer than the playback time of the video content are associated. As an example, when the playback time of the material is shorter than the playback time of the video content, it is associated that the material is looped for playback or a digest video is generated by editing the content shown in the insufficient time. Also, when the playback time of the material is longer than the playback time of the video content, it may be associated that a predetermined part (for example, the end) of the material is deleted. When the playback time of the material is different from the playback time of the video content associated with the template, the specifying unit 133 refers to the processing method for the insufficient or surplus time associated with the template and specifies the content to be corrected.

[0039] The generation unit 134 generates a prompt based on the content to be corrected specified by the specifying unit 133. As an example, when the playback time of the material is 9 minutes and the playback time of the video content is 10 minutes, and the processing method for the surplus time is the generation of a digest video, the generation unit 134 generates a prompt stating "Please generate a digest of the input content in 1 minute and connect the generated digest after the input content to generate a video." Also, when the playback time of the material is 10 minutes and the playback time of the video content is 60 minutes, and the processing method for the surplus time is loop playback, a prompt stating "Please generate a video that repeats the input content 6 times" is generated.

[0040] The information processing apparatus 1 may be configured to generate a prompt for correcting the content captured in the material that is a photographic image.

[0041] When the description of the subject of the photographic image indicated by the metadata does not match the description of the video content indicated by the generation instruction, the generation unit 134 generates a prompt for correcting the photographic image based on the description of the video content indicated by the generation instruction. As an example, when the metadata indicates that the content included in the image that is the material is "5 dumplings" and the generation instruction indicates that the content of the video content is "6 dumplings", the generation unit 134 generates a prompt for correcting the material to 6 dumplings.

[0042] The information processing apparatus 1 may be configured to generate a prompt that further includes restrictions when generating content as an instruction.

[0043] The generation instruction further includes a restriction item indicating a matter whose expression is restricted in the video content. The restriction item is, for example, to prevent a brand logo image or a character for copyright purposes from being included in the generated content. The restriction item may be, for example, that the content suggesting violence or crime is not included in the content. As an example, on the reception screen, an interface for selecting a restriction item to be specified for the generated content is arranged, and the reception unit 131 may receive the restriction item selected by the user on the reception screen as a generation instruction.

[0044] The generation unit 134 generates a prompt that further includes a restriction item. As an example, the generation unit 134 may generate a prompt configured such that a sentence indicating the restriction item is further continuous in the sentence specifying the content of the content.

[0045] [Processing flow in the information processing apparatus 1] FIG. 5 is a flowchart showing the processing flow in the information processing apparatus 1. The reception unit 131 receives a material and a generation instruction (S01). The acquisition unit 132 acquires metadata about the material received by the reception unit 131 (S02).

[0046] The specifying unit 133 specifies the correction content of the material based on the generation instruction and the metadata of the material (S03). The generation unit 134 generates a prompt for inputting an instruction to correct the material to the image generation AI based on the specified correction content (S04). Then, the information processing apparatus 1 ends the processing.

[0047] [Effect by Information Processing Apparatus 1] As configured above with the information processing apparatus 1, there is an effect of reducing the burden of correcting the material to conform to the content.

[0048] Note that the present invention can contribute to Goal 9, "Build the infrastructure for industry and technological innovation," of the Sustainable Development Goals (SDGs) led by the United Nations.

[0049] As described above, the present invention has been described using embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist. For example, all or part of the device can be configured by functionally or physically dispersing and integrating it in any unit. Also, new embodiments resulting from any combination of multiple embodiments are included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination have the effects of the original embodiments combined.

Explanation of Reference Numerals

[0050] 1 Information processing apparatus 2 Information terminal 3 Generation apparatus 11 Communication unit 12 Storage unit 13 Control unit 131 Reception unit 132 Acquisition unit 133 Specifying unit 134 Generation Unit 135 Determination Unit

Claims

1. a receiving unit that receives a material for constituting a video content and a generation instruction for generating the video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content and a description explaining a content expressed by the video content; an acquisition unit that acquires metadata, which is data describing the material, the metadata including an aspect ratio of the material; a specification unit that specifies a modification content for applying the material to the video content based on the generation instruction and metadata of the material; A generation unit that generates a prompt for inputting an instruction to modify the material to an image generation AI based on the identified modification content; having the identifying unit, when the aspect ratio indicated by the metadata is different from the reference aspect ratio, identifies a change in the aspect ratio of the material as the modification content; When the specification unit specifies a change in aspect ratio as the modification content, the generation unit generates, based on the generation instruction, a prompt for instructing to supplement a portion that is insufficient for changing the aspect ratio of the material based on the modification content, based on a description explaining the content expressed by the video content indicated by the generation instruction. Information processing device.

2. When the specification unit specifies a change in aspect ratio as the modification content, the generation unit generates, based on the generation instruction, a prompt for instructing to cut off a portion that would become redundant if the aspect ratio of the material is changed based on the modification content. The information processing device according to claim 1 .

3. said material includes photographic images; the metadata includes descriptions describing subjects of the photographic images; when a description describing a subject of the photographic image indicated by the metadata does not match a description describing a content expressed by the video content indicated by the generation instruction, the generation unit generates a prompt for modifying the photographic image based on the description describing the content expressed by the video content indicated by the generation instruction. The information processing device according to claim 1 .

4. a determination unit that determines whether or not the material needs to be modified to apply to the video content based on the metadata and the generation instruction, The identification unit identifies the content of the correction when the determination unit determines that the material needs to be corrected. The information processing device according to claim 1 .

5. The generation instruction further includes a restriction indicating a restriction on an expression of the video content, The generating unit generates a prompt further including the restriction. The information processing device according to claim 1 .

6. The generation unit generates a prompt by inputting the correction content identified by the identification unit into a prompt generation model that has been trained to output a prompt for inputting an instruction to correct the material to the image generation AI when a correction content for applying the material to the video content is input, The prompt generation model learns training data that associates a training prompt with an evaluation of an image generated by an image generation model based on the training prompt. The information processing device according to claim 1 .

7. The computer executes receiving a generation instruction for generating a video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content, and a description explaining a content expressed by the video content; obtaining metadata describing the material, the metadata including an aspect ratio of the material; determining, based on the generation instructions and metadata of the material, modifications to be applied to the material to the video content; generating a prompt for inputting instructions to the image generation AI to modify the material based on the identified modification content; having In the step of specifying, if the aspect ratio indicated by the metadata is different from the reference aspect ratio, a change in the aspect ratio of the material is specified as the modification content; In the generating step, when a change in aspect ratio is identified as the modification content in the identifying step, a prompt is generated based on the generating instruction to instruct the user to supplement a portion that is missing in order to change the aspect ratio of the material based on the modification content, based on a description explaining the content expressed by the video content indicated by the generating instruction. Information processing methods.

8. On the computer, receiving a generation instruction for generating a video content, the generation instruction including a reference aspect ratio that is a reference aspect ratio for the video content, and a description explaining a content expressed by the video content; obtaining metadata describing the material, the metadata including an aspect ratio of the material; determining, based on the generation instructions and metadata of the material, modifications to be applied to the material to the video content; generating a prompt for inputting instructions to the image generation AI to modify the material based on the identified modification content; Run the command, In the step of specifying, if the aspect ratio indicated by the metadata is different from the reference aspect ratio, a change in the aspect ratio of the material is specified as the modification content; In the generating step, when a change in aspect ratio is identified as the modification content in the identifying step, a prompt is generated based on the generating instruction to instruct the user to supplement a portion that is missing in order to change the aspect ratio of the material based on the modification content, based on a description explaining the content expressed by the video content indicated by the generating instruction. program.

Images