Information processing device, information processing method, and program

Abstract

To reduce the workload for users in obtaining production data. [Solution] The information processing device includes an acquisition means for acquiring the context of a prompt used to generate a template, along with the template; an input means for inputting the context acquired by the acquisition means into a generating AI (Artificial Intelligence); and an editing means for editing the production data using the template acquired by the acquisition means, using the generating AI into which the context has been input.

Description

Technical Field

[0004] , , , , , , ,

[0005] , , ,

[0001] This disclosure relates to prompt generation technology for generative AI.

Background Art

[0002] Software for creating data of productions such as posters and flyers is provided. As one of the functions of such software, there is a function in which a user selects a desired template from various prepared templates and inserts arbitrary characters or an image taken by the user himself / herself into the template. Patent Document 1 discloses a technique for searching for a template that harmonizes with an image based on the impression similarity obtained by determining the impression of the image as an input and the impression similarity of the template.

[0003] Also, services for generating templates for productions using generative AI (Artificial Intelligence) have been proposed. In this service, when a user inputs in words or sentences what kind of production they want to create, the generative AI automatically generates and outputs a template for the production based on the input words or sentences. Such words or sentences for instructing the generative AI to generate are called prompts. Also, software having a function of making an image generation AI create an image and inserting the image into a template has been proposed. An image generation AI is a generative AI that generates an image based on a text prompt such as words or sentences by inputting an image or atmosphere desired by the user.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, editing products created using generation AI required users to think of appropriate prompts and input them into the generation AI in order to obtain the desired product, which was a difficult and time-consuming task for the user. [Means for solving the problem]

[0006] The information processing device disclosed herein includes: an acquisition means for acquiring the context of a prompt used to generate a template, along with the template; an input means for inputting the context acquired by the acquisition means into a generating AI (Artificial Intelligence); and an editing means for editing the production data using the template acquired by the acquisition means, using the generating AI into which the context has been input. [Effects of the Invention]

[0007] According to the information processing device disclosed herein, the workload on the user to obtain the data of the created product can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of the information processing system according to this embodiment. [Figure 2] This figure shows an example of a server hardware configuration. [Figure 3] This figure shows an example of a client's hardware configuration. [Figure 4] This is a functional block diagram showing an example of the software configuration of an information processing system. [Figure 5] This is a flowchart showing the process of generating AI. [Figure 6] This figure shows an example of the dialogue history on a prompt input screen. [Figure 7] This figure shows the context generated based on the interaction history with the generating AI. [Figure 8] This is a flowchart showing the processing flow performed by the template creation application. [Figure 9]This flowchart shows the processing flow performed by the poster editing app. [Figure 10] This figure shows an example of the UI screen of a poster editing app. [Figure 11] This diagram illustrates the behavior when a template is selected. [Figure 12] This is a flowchart showing the content change process. [Figure 13] This diagram illustrates the screen transitions in the editing screen during the content modification process. [Figure 14] This flowchart shows the flow of the variation generation process. [Figure 15] This diagram illustrates how to change template variations. [Modes for carrying out the invention]

[0009] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. Note that the following embodiments are not intended to limit the scope of the claims of this disclosure, and not all combinations of features described in these embodiments are necessarily essential to the solutions of this disclosure.

[0010] One use case involves creating poster data by having a user generate an image using an image generation AI and then inserting that image into an existing template. In such cases, if the user does not provide accurate prompts to the image generation AI, the generated image may not match the template. For example, if the images included in the template are in a realistic style, and the user's prompts are not accurate, the image generation AI may generate an image that does not match the style and atmosphere of the template, such as an illustrative image. In such cases, the user would have to try entering different prompts and repeat the image generation process with the image generation AI until the desired image was produced.

[0011] Therefore, in the present embodiment, the context, which is a summary of the prompt used to generate the template to be edited, is associated with and stored together with the template. Then, when using the template in poster editing, the context associated with the template is retrieved and loaded (input) into the generation AI used for editing. This makes it easier to generate a poster that matches the style and atmosphere of the template.

[0012] In the following description, the "template" refers to information indicating the arrangement, color, size, font, etc. of the content (text and images) and graphics arranged in a production such as a poster. For example, one or more image setting objects, text setting objects, graphic setting objects, etc. are arranged in a region of a predetermined width. In addition to the arrangement position, size, and angle, each object is set with metadata necessary for generating the production. For example, as the metadata of the text setting object, what types of text information such as title, subtitle, body text, etc. are arranged is held. As the metadata of the image setting object, designation information of the image file, attribute information of the image, etc. are held. Also, as the metadata of the graphic setting object, the shape and color scheme of the graphic are held. In addition, information on the use and category of the template may be added to the template. In the present embodiment, the template data is in the form of HTML text, but it is not limited to this and may be in other forms.

[0013] "Image" includes, unless otherwise specified, still images and frame images extracted from moving images. In the following embodiments, a poster is described as an example of a "production", but the production is not limited to posters. Flyers, menus, banners, calendars, photo collages, certificates of commendation, certificates, business cards, shop cards, postcards, invitations, membership cards, and any other production that includes at least one of image content and text content can be used. Further, these productions can be used not only in printed form but also as electronic content on websites, SNS, virtual spaces, etc. The data of the production to be edited is referred to as production data.

[0014] In this embodiment, a template creation application that generates a template using a template generation AI and a poster editing application that edits a poster using the template generated by the template creation application will be described. In the following description, the application is abbreviated as an app.

[0015] <System Configuration> FIG. 1 is a diagram showing a configuration example of an information processing system 100 according to this embodiment. The information processing system 100 is composed of a server 101 that provides a content generation service and a client 102 that uses the service. The server 101 and the client 102 can be communicatively connected to each other via the Internet 104 from their respective networks 103. The server 101 has a backend app 105. The client 102 has an operating system 106, a template creation app 107, and a poster editing app 108. In this embodiment, a poster is taken as an example of the editing target of the poster editing app 108 for description, but the editing target is not limited to posters. For example, the technology of the present disclosure can also be applied to software for obtaining various productions such as photo album creation software and postcard creation software. Note that in FIG. 1, there is one server 101, but it may be composed of a plurality of server devices. Also, the number of clients 102 connected to the server 101 is not limited to one and may be plural.

[0016] Figure 2 shows an example of the hardware configuration of an information processing device that functions as a server 101. The server 101 is an information processing device that includes a CPU 201, ROM 202, RAM 203, HDD 204, network interface 205, user interface 206, etc. These parts are connected by a bus 207. The CPU 201 is a central processing unit that executes programs read from ROM 202, RAM 203, HDD 204, etc. ROM 202 is a non-volatile memory that stores the operating system, embedded programs, and data. RAM 203 is a volatile memory that provides a temporary memory area. HDD 204 is a mass storage device. HDD 204 may be flash memory, an SSD, or external storage connected via the network interface 205. The user interface 206 is an operation display unit that includes a display, keyboard, mouse, buttons, touch panel, etc., and performs input and output of information and signals. The network interface 205 connects to a network such as a LAN and communicates with other computers and network devices. The communication method may be either wired or wireless.

[0017] Figure 3 shows an example of the hardware configuration of an information processing device that functions as a client 102. The client 102 consists of a PC (personal computer), smartphone, tablet terminal, etc., and has a CPU 301, ROM 302, RAM 303, HDD 304, network interface 305, display unit 306, and input unit 307. These units are connected by a bus 309. The CPU 301 is a central processing unit that executes programs read from ROM 302, RAM 303, HDD 304, etc. ROM 302 is a non-volatile memory that stores the operating system 106, embedded programs, and data. RAM 303 is a volatile memory that provides a temporary memory area. HDD 304 is a mass storage device. HDD 304 may be flash memory, an SSD, or external storage connected via the network interface 305. HDD 304 stores various application programs such as the template editing application 107 or the poster editing application 108, and data necessary for processing.

[0018] The display unit 306 includes a display and a display control circuit, and displays the display data output from the CPU 301. The input unit 307 includes input devices such as a keyboard, mouse, buttons, and touch panel, and inputs information and signals from the user to the CPU 301. The network interface 305 connects to a network such as a LAN and communicates with other computers and network devices. The communication method can be either wired or wireless.

[0019] Note that the configurations shown in Figures 2 and 3 are examples, and it is not necessary to include all of the illustrated configurations. For example, it is possible to use a computer with a simplified configuration that does not have all of these hardware components, and to connect to another computer using functions such as remote desktop or remote shell, and operate it remotely to function as an information processing device. Also, for example, when using a smartphone as the terminal for client 102, it may include configurations other than those illustrated, such as a microphone for calling, a speaker, and a camera for imaging.

[0020] <Software Configuration> Figure 4 shows an example of the software configuration of the information processing system 100. The software of the information processing system 100 includes a template creation application 107, a backend application (hereinafter referred to as the backend application 105), and a poster editing application 108. In this embodiment, the template creation application 107 is installed on an information processing device (hereinafter referred to as the client 102) that functions as a client 102, and is operated by a template creator Ua. The poster editing application 108 is installed on the client 102 and is operated by a poster editor Ub, who is different from the template creator Ua. Note that the template creation application 107 and the poster editing application 108 may be installed on the same client 102, or on different clients 102. The template creation application 107 and the poster editing application 108 may be native applications, or they may be web applications that run in the browser of the client 102, and this disclosure is not limited to the form of the applications. The backend application 105 may be installed on the server 101, or it may be installed on the client 102. In this embodiment, the server 101 is equipped with the backend application 105.

[0021] <Backend application (generating AI)> The backend application 105 includes a template generation AI 400, a text generation AI 470, and an image generation AI 480. The template generation AI 400, text generation AI 470, and image generation AI 480 may be installed on a single server 101, or they may be installed on multiple servers 101. In addition, the template generation AI 400, text generation AI 470, and image generation AI 480 may utilize template generation services, text generation services, and image generation services provided by a service provider's server 101 via the internet.

[0022] Examples of generative AI models include language models such as LLM (Large Language Model) and SLM (Small Language Model), image generation models that generate images, or multimodal models that can handle both text and images. In language models, the input to the model is text, and the output is also text. In image generation models, the input to the model is text, and the output is an image. In multimodal models, the input to the model is either text and / or images, and the output is either text and / or images. The generative AI models discussed in this disclosure may consist of only a language model, or a language model and an image generation model that exist independently, or a multimodal model that contains a model that handles both text and images.

[0023] Examples of template generation AI 400 include Adobe Express Text to Template [https: / / new.express.adobe.com / ?category=generative-ai]. Examples of text generation services include ChatGPT [https: / / chatgpt.com / ] and GPT-4 [https: / / openai.com / index / gpt-4 / ]. Examples of image generation AI 480 include DALL-E [https: / / openai.com / index / dall-e-3 / ].

[0024] The template creation application 107 works in conjunction with the backend application 105, which consists of the template generation AI 400, text generation AI 470, and image generation AI 480, to perform template creation and context generation processes. The poster editing application 108 works in conjunction with the backend application 105, which consists of the template generation AI 400, text generation AI 470, and image generation AI 480, to perform poster data editing processes using the template. The template creation application 107 and the poster editing application 108 will be described later.

[0025] The template generation AI 400 receives prompts sent from the template creation application 107 or the poster editing application 108 as input and inputs them to the template generation unit 401. The template generation unit 401 performs generation processing based on the prompts and generates a template. The template generation AI 400 outputs the template generated by the template generation unit 401 to the template creation application 107 or the poster editing application 108. The template generation unit 401 is a model that has been trained for template generation. The prompts input to the template generation AI 400 are not limited to text; if it is possible to specify a reference image, that reference image may be included in the prompt.

[0026] The text generation AI 470 receives prompts sent from the template creation application 107 or the poster editing application 108 as input and inputs them to the text generation unit 402. The text generation unit 402 generates a text response to the prompt. For example, if the text prompt "Summarize document X" is input to the text generation AI 470, the text generation unit 402 generates a text summary of document X. The text generation AI 470 outputs the summary text generated by the text generation unit 402 as context to the template creation application 107 or the poster editing application 108. The text generation unit 402 is a language model that has been trained at least for context (summary) generation. The prompts input to the text generation AI 470 are not limited to text; if it is possible to specify a reference image, that reference image may be included in the prompt.

[0027] The image generation AI 480 receives prompts sent from the template creation application 107 or the poster editing application 108 as input and inputs them to the image generation unit 403. The image generation unit 403 generates an image as a response to the prompt. For example, if the text prompt "Create an image of a reindeer" is input to the image generation AI 480, the image generation AI 480 will generate an image of a reindeer. The style and atmosphere of the image can be specified by the prompt. The image generation AI 480 outputs the image generated by the image generation unit 403 to the template creation application 107 or the poster editing application 108. The image generation unit 403 is an image generation model that has been trained for image generation. The prompts input to the image generation unit 403 are not limited to text; if a reference image can be specified, that reference image may be included in the prompt.

[0028] In this embodiment, the template creation application 107 or the poster editing application 108 accepts prompt input to the generation AI in an interactive manner. The dialogue history of the prompt input is then stored. This dialogue history is subject to context summarization, which will be described later.

[0029] In other words, the prompts subject to context summarization include the prompts entered by the user and the responses returned by the generating AI to these prompts (hereinafter referred to as "the generating AI's responses"). The prompts entered by the user are words or sentences used to instruct the generating AIs, such as the template generating AI 400, text generating AI 470, and image generating AI 480, on what to generate. The prompts entered by the user themselves will be referred to as the first prompt below.

[0030] Furthermore, the prompt entered by the user may include text that has been converted from the first prompt in a way that is easy for the generating AI to understand. Conversion means transforming the prompt entered by the user into simpler phrasing or specific instructions for the generating AI. Hereafter, the converted first prompt will be referred to as the second prompt. For example, if the user enters the sentence "Summarize document X" as a prompt (first prompt) to the text generating AI 470, the template creation app 107 or poster editing app 108 converts it to a second prompt such as "Summarize document X in bullet points" and enters the second prompt into the text generating AI 470.

[0031] Furthermore, when using a generative AI with a multimodal model that can handle not only text but also images, images can be included in the prompt. For example, if an image file named "Image Y" and the text "Please describe Image Y" are input to the generative AI as prompts, the generative AI will generate and output text that describes the content of Image Y. In this case, the image input as a prompt can be included in the context summary.

[0032] The responses from the Generating AI included in the context summarization are the text and images output by the Generating AI. Specifically, in a dialogue with Template Generating AI 400, the Generating AI's response includes text or an image representing the template. The text representing the template is the HTML text if the template is written in HTML format. The image representing the template is the image of the template converted into an image file format.

[0033] <Template creation app> As shown in Figure 4, the template creation application 107 includes a prompt receiving unit 410, a template acquisition unit 411, a context acquisition unit 412, and a storage unit 413. The template creator Ua launches the template creation application 107 on the client 102 and uses the template creation application 107 to create the desired template.

[0034] The prompt receiving unit 410 receives prompt input from the user. In this embodiment, the prompt receiving unit 410 receives prompt input in a dialogue format between the user and the generating AI. The dialogue format is a format in which the user inputs a request to the generating AI in natural language containing one or more words, and the processing result of the generating AI in response to that request is output as an answer. The dialogue may be repeated. The series of dialogue history is stored in a predetermined memory area such as the RAM of the client 102. Note that prompt input is not limited to the dialogue format, and may also be an input method according to a predetermined input form. The prompt may be text, or may include text and images. Also, the language of the prompt is not limited to a language used in a specific country such as Japanese or English, but may be any language. Prompt input will be described later.

[0035] The template acquisition unit 411 works in conjunction with the template generation AI 400 to acquire one or more templates generated by the template generation AI 400. That is, the template acquisition unit 411 sends the prompt received from the user by the prompt reception unit 410 to the template generation AI 400 and requests the generation of a template. In response to the request, the template generation AI 400 generates a template using the prompt as input and outputs it to the template acquisition unit 411. The template acquisition unit 411 may also convert the prompt entered by the user (first prompt) received by the prompt reception unit 410 in a way that is easy for the template generation AI 400 to understand, and send the converted prompt (second prompt) to the template generation AI 400. In this embodiment, the template acquisition unit 411 is shown as acquiring the template as HTML text data, but other data formats may also be used.

[0036] The context acquisition unit 412 acquires the context of the prompts used to generate the template acquired by the template acquisition unit 411. As described above, the context summary targets the prompts used to generate the template and the generating AI's response to those prompts. If the template acquisition unit 411 receives prompts in a conversational format, the entire conversation (conversation history) up to obtaining the template is included in the summary. Specifically, as described above, the context summary targets include at least one of the following: the first prompt, which is a user prompt received by the prompt reception unit 410; the second prompt, which is the first prompt converted into a format suitable for input to the generating AI; and the generating AI's response. These prompts may be text or may include images.

[0037] The context acquisition unit 412 acquires the context as text data. The context acquired by the context acquisition unit 412 also includes the template data, which is the response of the generating AI acquired by the template acquisition unit 411. Furthermore, the context acquisition unit 412 may acquire the context by excluding prompts or responses of the generating AI that are not reflected in the final confirmed (acquired) template from the dialogue history to be summarized.

[0038] The context acquisition unit 412 includes the above-mentioned summarization target in the prompt and requests the text generation AI 470 to generate a summary, and acquires the processing result of the text generation AI 470 as context. The text generation AI 470 may update its model based on the prompt input from the context acquisition unit 412. The context acquisition unit 412 may also acquire the context using a predetermined algorithm instead of the text generation AI 470. The process of generating and acquiring the context will be described later.

[0039] The memory unit 413 stores the context acquired by the context acquisition unit 412 in a predetermined memory area, associating it with the template acquired by the template acquisition unit 411. The predetermined memory area may be the HDD 304 of the information processing device (client 102) where the template creation application 107 is deployed, the HDD 204 of the server 101, or cloud storage, etc. The memory unit 413 may also store the context acquired by the context acquisition unit 412 in association with the category to which the template belongs. For example, if the template acquisition unit 411 generates multiple templates related to Christmas from the same prompt (dialogue history), the memory unit 413 stores these multiple templates as a single category. Furthermore, the memory unit 413 stores the context of the dialogue history in association with that category.

[0040] <Poster editing app> As shown in Figure 4, the poster editing application 108 includes an acquisition unit 421, a context input unit 422, an editing unit 423, and a context update unit 426. The editing unit 423 includes a content editing unit 424 and a template editing unit 425. The poster editor Ub launches the poster editing application 108 on an information processing device that functions as a client 102, and edits the poster by editing the template created by the template creation application 107 using the poster editing application 108.

[0041] The acquisition unit 421 acquires the context of the prompt used to generate the template, along with the template itself. The template is the template used as the basis for the poster data to be edited, and is selectable by the user. The context and template are pre-stored in the client 102 or server 101 that runs the poster editing application 108.

[0042] The context input unit 422 inputs (loads) the context acquired by the acquisition unit 421 to the generation AI. The generation AI targeted for input is the generation AI used for editing posters. In this embodiment, the template generation AI 400 and the image generation AI 480 are loaded with the context. In this way, the context is pre-loaded as prompts for these generation AIs. As described above, the context includes the prompts and responses (dialogue history) used when the template was generated.

[0043] The editorial unit 423 uses a generation AI with context input to edit the poster data using the template acquired by the acquisition unit 421. Editorial unit 423 accepts additional prompt input from the user, inputs the additional prompt to the generating AI which has the context entered, and obtains the modified template including the results of processing by the generating AI. The modified template is output as edited poster data. Editorial Department 423 works in conjunction with generation AIs such as template generation AI 400, text generation AI 470, and image generation AI 480 to edit templates and the content contained within them. Editorial Department 423 requests these generation AIs to generate content such as text and images, or to generate templates, or to modify variations.

[0044] The content editing unit 424 performs a process to modify the content included in the template acquired by the acquisition unit 421 (hereinafter referred to as the content modification process). In the content modification process, the content editing unit 424 edits the content, such as images and text, included in the template. Content editing includes changing images and changing text. The content modification process will be described later.

[0045] The template editing unit 425 executes a process (hereinafter referred to as the variation generation process) to automatically generate variations of the template acquired by the acquisition unit 421. The variation generation process will be described later. The template editing unit 425 also edits the template according to the operations of the poster editor Ub. Editing a template includes changing the content (images, text) included in the template, changing the placement of content, changing colors and fonts, etc.

[0046] The context update unit 426 updates the context acquired by the acquisition unit 421 to a new context that includes the additional prompt received by the editorial unit 423, and stores it linked to the poster data. The process of updating the context is the same as the process in the context acquisition unit 412. That is, the context update unit 426 requests the text generation AI 470 to generate a summary, using the context acquired by the acquisition unit 421, the additional prompt received by the editorial unit 423, and the generation AI's response to the additional prompt as the subject of the summary, and acquires the processing result of the text generation AI 470 as the context. Note that the context acquisition unit 412 may update the context using a predetermined algorithm instead of the text generation AI 470.

[0047] A template creator (Ua) can be either a service provider offering the poster editing app 108 service, or an end-user who creates and edits a poster from scratch without using a template provided by the service provider. In the former case, the template creator (Ua) belongs to the service provider, and the service provider creates a template using the template creation app 107. The service provider then provides the created template to the poster editing app 108. In the latter case, the template creator (Ua) belongs to the end-user. The end-user creates a template using the template creation app 107. The end-user then edits the template they created using the poster editing app 108.

[0048] The software configuration example in Figure 4 describes a case where the template creation application 107 and the poster editing application 108 are provided as separate application programs, but they may also be provided as a single application program. That is, a single application program may include both template creation and poster editing functions. For example, if the template creator Ua is an end user and performs both template creation and poster editing, they may be combined into a single application.

[0049] Furthermore, the example in Figure 4 illustrates a configuration in which generation AIs such as template generation AI 400, text generation AI 470, and image generation AI 480 are included in the backend application 105 and not included in the template creation application 107 or poster editing application 108. However, this disclosure is not limited to this configuration. These generation AIs may also be included in the template creation application 107 or poster editing application 108.

[0050] <Processing of generated AI> Next, we will explain the processing flow performed by the generating AI. Figure 5 shows the processing flow of the generated AI. In the following explanation, the symbol "S" represents a step.

[0051] Figure 5(a) is a flowchart showing the processing flow executed by the template generation AI 400. The CPU 201 of the server 101 reads a program containing the processing shown in Figure 5(a) from the ROM 202 or HDD 204, loads it into RAM 202, and executes the processing shown in the flowchart. The same applies to the following flowcharts. The processing shown in the flowchart of Figure 5(a) is started when the server 101 receives a template creation request from the client 102.

[0052] In S501, the template generation AI400 receives prompts from the template creation app 107 or the poster editing app 108.

[0053] In S502, the template generation AI 400 generates a template from the received prompt. Template generation is performed by the template generation unit 401 of the template generation AI 400.

[0054] In S503, the template generation AI400 sends the generated template to the requesting template creation application 107 or poster editing application 108.

[0055] The method for generating templates in S502 will be explained. For example, a large amount of training data is prepared in advance, with a template written in HTML format as the target variable and the text representing that template as the explanatory variable, and a model is created in advance by training on this data using a deep learning method. This deep learning model functions as the template generation unit 401 of the template generation AI 400. The template generation AI 400 receives a prompt from S501 as input to its model, and generates and outputs a template corresponding to that prompt. In this embodiment, the template is described using HTML format text as an example, but this disclosure is not limited to this, and other formats of text or images may also be used. The template generation AI 400 may generate and output multiple templates for a single input (prompt).

[0056] Figure 5(b) is a flowchart showing the flow of the context generation process performed by the text generation AI 470. The process shown in the flowchart of Figure 5(b) starts when the server 101 receives a context generation request from the client 102.

[0057] In S511, the text generation AI 470 receives a prompt from the template creation application 107 or the poster editing application 108. The prompt may be something like "Summarize document X". Document X is text, and in this embodiment, for example, it is the content of a dialogue (dialogue history) that includes the prompt entered by the user and the response from the generation AI in the template creation application 107 or the poster editing application 108.

[0058] In S512, the text generation AI 470 generates a text summary (context) from the content of the prompt received in S511. The text summary is performed by the text generation unit 402 of the text generation AI 470.

[0059] Here, we will explain the method of summarizing text. For example, a large amount of training data is prepared in advance, with a certain text as the explanatory variable and a summarized text of that text as the target variable, and a model is created by training a large amount of this data using a deep learning method. This deep learning model functions as the text generation unit 402 of the text generation AI 470. The text generation AI 470 generates the summarized text by inputting the text to be summarized, received in S511, as a prompt to the model. In this embodiment, the method of summarization is described using deep learning, but this disclosure is not limited to this and other methods may be used. For example, the process may involve extracting strings from the text to be summarized according to specific rules, or rearranging the extracted strings into a predetermined format.

[0060] Here, we will explain the purpose of summarizing text. The main purposes of summarizing text are as follows: 1. The purpose is to shorten the text. 2. The purpose is to convert the text into a format that is easy for the generating AI to understand. 3. The purpose is to remove content that does not ultimately conform to what the AI ​​generates.

[0061] First, let's explain the first purpose of shortening the text. In this disclosure, the context is input (read) into the template generation AI 400 by the poster editing application 108. If the number of characters in the text read into the template generation AI 400 is long, the processing time of the template generation AI 400 will increase, which may cause a delay in response to the user. Also, if the generation AI used by the poster editing application 108 is a paid service, there may be a cost depending on the length of the prompt (text) input into the generation AI. Therefore, by using a context with a shortened text length through summarization as the prompt for the generation AI used by the poster editing application 108, the problems of response delay and cost can be reduced.

[0062] Let's explain the second reason: "the purpose of converting to text that is easy for the generating AI to understand." Generally, prompts are more likely to produce the user's expected answer if they are formatted in a way that is easy for the generating AI to understand, such as by including conditions and objectives with # (hash symbols) or by using bullet points. Therefore, it is preferable to summarize the prompt so that it is easy for the template generating AI 400 to understand. An example of conversion is using a bulleted list format (Figure 7, 711).

[0063] The third purpose is to remove content that does not match the final output generated by the AI. When creating a template using the template creation application 107, the interaction between the user and the template generation AI 400 is often repeated through trial and error. Therefore, the interaction history may contain dialogue that is not very relevant to the final template generated by the template generation AI 400. If the template generation AI 400 is fed dialogue that is not very relevant to the template it has generated, there is a high possibility that it will generate content that is not suitable for the template. For this purpose, it is preferable that the content of dialogue that does not match the final template generated by the template generation AI 400 is removed from the prompts used when editing this template. In this case, the context acquisition unit 412 of the template creation application 107 instructs the text generation AI 470 to create a summary by removing the content of dialogue that does not match the final template from the interaction history to be summarized.

[0064] In S513, the text generation AI 470 sends the context generated by the text generation unit 402 to the requesting template creation application 107 or poster editing application 108.

[0065] Figure 5(c) is a flowchart showing the processing flow performed by the image generation AI 480. The processing shown in the flowchart of Figure 5(c) begins when the server 101 receives an image generation request from the client 102.

[0066] In S521, the image generation AI480 receives prompts from the template creation application 107 or the poster editing application 108. In S522, the image generation AI 480 generates an image from the prompt received in S521. Image generation is performed by the image generation unit 403 of the image generation AI 480. In S523, the image generation AI 480 sends the image generated in S522 to the requesting template creation application 107 or poster editing application 108.

[0067] The method for generating images in S522 is described below. For example, a large amount of training data is prepared in advance, with an image as the target variable and text describing that image as the explanatory variable. A model is then created by training on this data extensively using a deep learning method. This deep learning model functions as the image generation unit 403 of the image generation AI 480. The image generation AI 480 receives a prompt (text) from S521 as input to the model, and generates and outputs an image appropriate to that prompt (text).

[0068] <Processing of the template creation application> The following describes the processing of the template creation application 107. First, we will describe the user interface screen (UI screen) of the template creation application 107.

[0069] Figure 6 shows an example of a prompt input screen 600 included in the UI screen of the template creation application 107. When the template creation application 107 is launched on client 102, the template creation application 107 displays the prompt input screen 600 shown in Figure 6 on the display unit 306. On the prompt input screen 600, the template creation application 107 interactively takes prompt input from the user and displays the response from the template generation AI 400. In the example in Figure 6, "Human" is displayed in the user input field and "AI" is displayed in the AI ​​response field. The areas indicated by symbols 601, 603, 605, and 607 are the user input fields, and the areas indicated by symbols 602, 604, 606, and 608 are the AI ​​response fields. Note that Figure 6 is written in English, but it may be in Japanese or other languages.

[0070] First, the template creator Ua enters the prompt "Create a template for the cafe opening poster in HTML format" into the user input field of the prompt input screen 600. This prompt is then sent to the template generation AI 400. The template generation unit 401 of the template generation AI 400 generates a template corresponding to that prompt. Since the prompt specifies that the template should be output in HTML format, the template generation unit 401 of the template generation AI 400 generates an HTML template. The generated template data (HTML text) is displayed in the AI ​​response field.

[0071] The template data, written in HTML text, can be visualized as a template containing text and image setting objects, where the colors, fonts, sizes, and layouts specified in HTML are expressed when displayed in a browser. Suppose the template creator Ua checks this template in a browser and decides to change the font of the text used in the template to Gothic. In that case, the template creator Ua enters the prompt "Use Gothic font for text" into the user input field. Then, the template generation unit 401 of the template generation AI 400 generates an HTML template with the font specified as Gothic. Note that the input to the template generation AI 400 can be just "Use Gothic font for text" entered in the user input field, as shown in Figure 6, or it can also include the past dialogue history (601 and 602) and send "Use Gothic font for text" (603). If only "Use Gothic font for text" (603) is sent, the server 101 may be configured to store the past dialogue history so that the template generation AI 400 can provide a response based on that history. If the past dialogue history (601 and 602) is also sent, the template generation AI 400 can provide a response based on that history even if the server 101 does not store the past dialogue history. Whether or not to send the past dialogue history is the same for subsequent dialogues.

[0072] Next, suppose template creator Ua wants to create an illustrative template. Template creator Ua enters the prompt "Create an illustrative template" into the user input field. Then, template generation unit 401 of template generation AI 400 generates an illustrative template in HTML text format and outputs it to the AI ​​response field. In this example, template generation unit 401 generates an illustrative template by specifying an illustrative image (cafe_illustration_image.jpg) in the HTML img tag. Here, it is assumed that template generation unit 401 obtains the illustrative image (cafe_illustration_image.jpg) by searching for it on a stock photo service.

[0073] Next, suppose template creator Ua wants to change the illustration-style image included in the answer template from an existing stock photo to an image created by template generation AI 400. Template creator Ua enters the prompt "Please insert a cafe image created by generative AI" into the user input field. Then, template generation unit 401 of template generation AI 400 requests image generation unit 403 to generate a cafe image. This process can be done, for example, by template generation unit 401 creating a prompt such as "create a cafe image" and calling image generation unit 403 with that as input to generate the image. Note that image generation unit 403 may call image generation AI 480 with only "create a cafe image" as input, or it may call image generation AI 480 with text including past dialogue history (601~607) in addition to "create a cafe image" as input. When the image generation AI 480 is called with only "create a cafe image" as input, if the server 101 stores the past dialogue history (601-607) and the image generation AI 480 is configured to read that information, an image based on the past dialogue history will be generated. On the other hand, when the image generation AI 480 is called with text that includes not only "create a cafe image" but also the past dialogue history (601-607) as input, an image based on the past dialogue history will also be generated. In this example, since the dialogue history includes a prompt (605) that generated an illustration-style image, even if the prompt does not explicitly include an illustration-style instruction like "create a cafe image", the image generation AI 480 will generate an illustration-style cafe image. As a result, the image generation unit 403 generates an illustration-style cafe image and returns that image to the template generation unit 401. The template generation unit 401 generates HTML-formatted text (template) specifying the image generated by the image generation unit 403 with an img tag and outputs it as the answer.Here, the filename of the image created by the image generation AI 480 is set to cafe_illustration_ai_image.jpg. In this way, the template creator Ua creates the desired template by repeatedly interacting with the template generation AI 400. In this example, the template generation AI 400 is shown generating HTML-formatted text (template), but this disclosure is not limited to that. The template generation AI 400 may generate templates using other formats of text, or it may generate templates that include image files. It may also generate templates in image file format.

[0074] Figure 7 shows an example of the prompt input screen 700. The prompt input screen 700 shows a continuation of the dialogue history (601-608) from the prompt input screen 600 shown in Figure 6. Context creation is achieved by the context acquisition unit 412 of the template creation application 107 performing the following processing. That is, the context acquisition unit 412 of the template creation application 107 acquires the context by summarizing the dialogue history (601-608) with the template generation AI 400. The context is text that shows a summary of the dialogue history, including the prompts and answers used to generate the template. The context may also be text that explains the content of the template created by the dialogue. In the example in Figure 7, the context is text that summarizes the input to the template generation AI 400 and the answers from the template generation AI 400.

[0075] The context acquisition unit 412 inputs "Summarize your history of human-AI interaction" as a prompt 710 to the prompt input screen 700 in order to create a context from the dialogue history (601-608) including these inputs and responses. This prompt 710 instructing the user to summarize may be manually entered by the template creator Ua, or it may be automatically generated and entered by the context acquisition unit 412 in response to a specific operation by the user. A specific operation could be, for example, pressing the OK button to confirm the template. The context acquisition unit 412 sends the dialogue history (601-608) and the prompt 710 instructing the user to summarize to the text generation AI 470. When the text generation AI 470 receives the dialogue history (601-608) and the prompt 710 instructing the user to summarize sent from the context acquisition unit 412 (template creation application 107), it inputs the received text into the text generation unit 402. The text generation AI 470 obtains summarized text as a result of processing by the text generation unit 402. This summarized text is the context 711. In the example in Figure 7, the context is expressed in bullet points and includes the following elements.

[0076] • An HTML template for advertising the opening of a cafe (a poster template in HTML format with the following features) • Illustrated design • Create images using generative AI. • The text is in a Gothic font.

[0077] The text generation AI 470 returns the context 711 generated by the text generation unit 402 to the requesting context acquisition unit 412 (template creation application 107). In this way, the context acquisition unit 412 acquires the context corresponding to the generated template. The context acquisition unit 412 stores the acquired context in the HDD 204, associating it with the finally generated template. The finally generated template is the template of the last answer generated immediately before the context 711 was generated, and in the example in Figure 7, it is the template corresponding to the generation AI's answer (608).

[0078] In this embodiment, a context was created using the text generation AI 470, but this disclosure is not limited thereto, and existing summarization algorithms may be used without using a generation AI such as the text generation AI 470. Also, in the above example, the dialogue history and a prompt instructing its summarization were sent as input to the text generation AI 470, but this disclosure is not limited thereto. For example, if the template generation AI 400 has stored past dialogue history, only the prompt instructing the summarization may be sent to the text generation AI 470. Also, in the above example, the template creator Ua manually entered the prompt instructing the summarization, but this disclosure is not limited thereto. A context creation button may be provided on the prompt input screen 600, and after the template creator Ua confirms the desired template, the template creation application 107 may, as an internal process, send a prompt instructing the text generation AI 470 to summarize the dialogue history up to that point by pressing the button. Also, in this disclosure, the summarized dialogue history is used as the context, but this disclosure is not limited thereto, and the unsummarized dialogue history itself may be loaded into the text generation AI 470 as the context.

[0079] As shown in Figure 7, the dialogue between the template creator Ua and the template generation AI400 rarely ends in a single interaction; it often involves multiple dialogues with trial and error. Some of these dialogues are not adopted into the final template. Since the content of these unadopted dialogues does not represent the final generated template, they are not suitable as elements that constitute the context. Therefore, as a prompt for generating the context, it may be necessary to instruct the AI ​​to exclude the dialogue history that was not reflected in the final template. For example, one could instruct the AI ​​to "Summarize your history of human-AI interaction. However, when you create the summary, exclude conversation histories that were not ultimately included in the template." Alternatively, among the template generation AI400's responses (602, 604, 606, 608), only the response corresponding to the final generated template (608) may be included in the context summary.

[0080] Figure 8 is a flowchart showing the processing flow executed by the template creation application 107. This flowchart begins when the template creation application 107 is launched on the client 102. The processing shown in this flowchart is written in a program stored in the ROM 302 or HDD 304 of the client 102. The program is called by the CPU 301 of the client 102, loaded into RAM 303, and executed by the CPU 301. In the following explanation, the symbol "S" represents a step.

[0081] In S801, the prompt receiving unit 410 of the template creation application 107 receives prompt input from the template creator Ua. The prompt receiving unit 410 displays the prompt input screen 600 shown in Figure 6 on the display unit 306 of the client 102 and accepts text input, which is the prompt, via the input unit 307. If the client 102 is capable of accepting text input via voice, it may also accept prompt input via voice. In this embodiment, the prompt is shown as an example of text input, but if the template generation AI 400 is capable of accepting it, it may also accept prompt input including images. In that case, the prompt receiving unit 410 accepts the specification of an image from a predetermined image folder or the search for an image from a website.

[0082] In S802, the template acquisition unit 411 of the template creation application 107 sends a template generation request to the template generation AI 400 of the server 101, along with the prompt entered in S801. When the template generation AI 400 receives the request from the client 102, it generates a template using the prompt received along with the request. The template generation AI 400 then sends the generated template data to the client 102. Note that when the template acquisition unit 411 sends a template generation request to the template generation AI 400, it may send the prompt entered in S801 as is along with the generation request, or it may convert the prompt entered in S801 into an input format suitable for the template generation AI 400 and send it along with the generation request. The template acquisition unit 411 stores the prompts sent to the template generation AI 400 as dialogue history in RAM 303.

[0083] In S803, client 102 receives template data generated by template generation AI 400 from template generation AI 400. The template acquisition unit 411 of template creation application 107 displays the received template data on display unit 306. As mentioned above, if the template is generated in HTML format, the template data is HTML format text. The template acquisition unit 411 displays the HTML format text in the AI ​​response field of the prompt input screen. The template acquisition unit 411 also displays the received template data. At this time, the template acquisition unit 411 may display the text in the AI ​​response field of the prompt input screen 600, or it may display it as a template with text and images arranged using the browser's functions. The template acquisition unit 411 adds the received template data (HTML format text) to the dialogue history and stores it in RAM 303.

[0084] In S804, the template acquisition unit 411 determines whether or not the template desired by the template creator Ua has been generated. For example, if an operation button such as an OK button is provided on the prompt input screen 600 and the user operates the OK button, the template acquisition unit 411 determines that the template desired by the template creator Ua has been generated. Otherwise, the template acquisition unit 411 determines that the template desired by the template creator Ua has not been generated. If the template desired by the template creator Ua has not been generated (S804; NO), the processing in S801 to S803 is repeated.

[0085] If the user clicks the OK button and it is determined that the template desired by the template creator Ua has been generated (S804; YES), proceed to S805.

[0086] In S805, the context acquisition unit 412 of the template creation application 107 sends a context generation request to the text generation AI 470 of the server 101. When requesting context generation, the context acquisition unit 412 requests the text generation AI 470 of the server 101 to summarize the prompt "Summarize your history of human-AI interaction" and the data held in RAM 303 as dialogue history, using these as the data to be summarized. The dialogue history includes at least one of the following: the prompt entered by the user in S801 (first prompt), the prompt converted in S802 (second prompt), and the template data which is the response of the generation AI acquired in S803. When the text generation AI 470 receives the request from the client 102, it summarizes the data to be summarized received along with the request and generates a context. The text generation AI 470 sends the generated context to the client 102.

[0087] In S806, client 102 receives the context generated by text generation AI 470. The context acquisition unit 412 of the template creation application 107 acquires the received context. The context acquisition unit 412 displays the acquired context on the prompt input screen 600 or another UI screen.

[0088] In S807, the storage unit 413 of the template creation application 107 associates the context obtained in S806 with the template obtained in S803 and saves it to the HDD 304. The template associated with the context is the template that was determined in S804 to be the desired template for the template creator Ua.

[0089] Let's further explain the memory unit 413 of the template creation application 107. The memory unit 413 stores the templates generated by the template generation AI 400 and the contexts generated by the text generation AI 470 in association with each other. Using the example in Figure 7, the HTML-formatted text (template data) finally obtained in the AI ​​response field 608 is stored in association with the context 711. The storage location can be the HDD 304 of the client 102, the HDD 204 of the server 101, or any area that can store data, such as cloud storage or a database accessible from the client 102. The memory unit 413 may store one context associated with one template, or it may store one context associated with multiple templates.

[0090] Furthermore, templates are categorized, and multiple templates may belong to a single category. For example, the "Christmas" category may contain multiple Christmas-related templates. Template creator Ua may interact with template generation AI 400 to create templates for Christmas-themed crafts and request the generation of multiple Christmas-themed craft templates. In this case, multiple templates will be generated for a given dialogue history. However, since a single context is generated from the series of dialogue histories, multiple templates will correspond to that single context. In this case, the memory unit 413 stores a single category associated with the multiple generated templates, and also associates a single context with these multiple templates. In other words, a context is associated with a category. In the editing process described later, the poster editing application 108 allows the selection of templates by specifying a category. For example, if poster editor Ub selects a template in the Christmas category, the context associated with the Christmas category, along with the multiple templates stored in association with that context, is called up and input to template generation AI 400.

[0091] <Processing in poster editing app> Figure 9 is an example flowchart showing the processing flow executed by the poster editing application 108. This flowchart begins when the poster editing application 108 is launched on the client 102. The processing shown in this flowchart is written in a program stored in the ROM 302 or HDD 304 of the client 102. The program is called by the CPU 301 of the client 102, loaded into RAM 303, and executed by the CPU 301. In the following explanation, the symbol "S" represents a step.

[0092] In S901, the poster editing application 108 displays the editing screen 910 on the display unit 306 of the client 102.

[0093] Figure 10 shows an example of the editing screen 1010, which is one of the UI screens 1000 of the poster editing application 108. The editing screen 1010 has a template list area 1021, a template editing area 1022, and a content editing area 1023. In S901, the template list area 1021 of the editing screen 1010 displays one or more thumbnails of templates that can be used to edit a poster. In the example of Figure 10, template thumbnails 1031, 1032, and 1033 are displayed. In this embodiment, these templates are assumed to be templates created by the template creation application 107. Although not shown, an operation area for selecting the size and category of the poster to be edited may also be displayed on the editing screen 1010.

[0094] In S902, the poster editing application 108 accepts the selection of a template. If any of the thumbnails displayed in the template list area 1021 is selected, the poster editing application 108 determines that the template corresponding to that thumbnail has been selected and proceeds to S903. If no thumbnails are selected, it waits for the selection of a template. Here, it is assumed that the poster editor Ub selects the template thumbnail 1032.

[0095] In S903, the poster editing application 108 retrieves the template 1052 corresponding to the selected thumbnail 1032 from a storage area such as the HDD 304 where the template 1052 is stored, and displays it as an editing target in the template editing area 1022. The poster editing application 108 also displays the selected thumbnail 1032 as selected 1042. The editing target template 1052 contains, for example, an image setting object 1061 for poster editor Ub to set an arbitrary image, and a text setting object 1062 for setting arbitrary text. The image setting object 1061 and the text setting object 1062 are shown in the size and position specified by the data (HTML format text) of the template 1052. Note that the number, position, size, and images and text within the image setting objects 1061 and text setting objects 1062 differ for each template.

[0096] Figure 11 illustrates the operation when template 1052 is selected by poster editor Ub. Figure 11(a) shows an example of the editing screen 1110 when template 1052 is displayed in the template editing area 1022. Template 1052 is a template created in advance by template creator Ua using the template creation application 107. Specifically, template 1052 is a template for a poster advertising the opening of a cafe, corresponding to the HTML format template data (AI response field 608) created using the template generation AI 400 based on the dialogue history shown in Figure 6.

[0097] As shown in Figure 11(a), template 1052 contains an image of a coffee cup 1181 and the text "Cafe Open" 1182. The image of the coffee cup 1181 is displayed in the image setting object 1061, and the text "Cafe Open" 1182 is displayed in the text setting object 1062.

[0098] In S904, the poster editing application 108 obtains the context associated with the template selected in S903. The context associated with a template is the one saved in S807 of the process executed by the template creation application 107. For example, if the cafe template 1052 is selected by the poster editor Ub on the editing screen 1110 in Figure 11(a), the poster editing application 108 obtains the context 711 associated with that template 1052 along with the template 1052. The context 711 is generated based on the dialogue between the generation AI and the template creator Ua when the template creation application 107 generates the template 1052. The association between the template and the context is performed by the storage unit 413 when the template 1052 is created. The poster editing application 108 may also display the obtained context on the editing screen 1110.

[0099] In S905, the poster editing application 108 loads the context acquired in S904 into the template generation AI 400.

[0100] Figure 11(b) shows the contents of the acquired context 711 and the input destination of the context 711. In this embodiment, the poster editing application 108 sends the acquired context 711 to either the template generation AI 400 or the image generation AI 480, or both, and causes the receiving template generation AI 400 or image generation AI 480 to read (input) the context. There are two methods for causing the template generation AI 400 and image generation AI 480 to read (input) the context: one is to store the context in the server 101 and have it read, and the other is to store the context in the client 102 and have it read. These two methods will be explained below.

[0101] In the first method, where the server 101 stores and loads the context, a storage area is provided in the backend application 105 of the server 101, or in the generation AI such as the template generation AI 400 or the image generation AI 480. When the template generation AI 400 or the image generation AI 480 receives a request to retrieve the context from the client 102, it loads the context from that storage area.

[0102] In the second method, where the client 102 stores and reads the context, the template creation application 107 or the poster editing application 108 is provided with a memory area for storing the context. When the template creation application 107 or the poster editing application 108 sends a prompt to the template generation AI 400 or the image generation AI 480, it retrieves the context from that memory area and sends it to the template generation AI 400 or the image generation AI 480 on the server 101. The template generation AI 400 or the image generation AI 480 on the server 101 reads the received context.

[0103] Here, the two examples described above are given as methods for loading the context into the template generation AI 400 or the image generation AI 480, but this disclosure is not limited to these. For example, a memory area for storing the context may be provided in the model of the generation AI, such as the template generation AI 400 or the image generation AI 480 (template generation unit 401, image generation unit 403, etc.), and when a request for context generation is received from the client 102, the generation AI may read the context from the model's memory area.

[0104] In this way, by loading context into generating AIs such as template generation AI 400 and image generation AI 480, the generating AI can understand the nature of the target template. For example, context 711 contains the text "illustrative design," so a generating AI that has read this context 711 will generate illustration-style content even if it is not explicitly instructed to create an "illustrative" template in the prompt.

[0105] Furthermore, in addition to loading the context into the generating AI such as template generation AI 400 or image generation AI 480, the model of the generating AI may also be updated (fine-tuned) based on the loaded context. That is, the model of the generating AI such as template generation AI 400, image generation AI 480, and text generation AI 470 is updated using the interaction history and context between the user and the generating AI during template editing using the poster editing application 108.

[0106] In S906, the poster editing application 108 displays a variation change button 1170 on the editing screen 1110. The button labeled "Change Variation" is the variation change button 1170. In the diagram, the button and text are shown in English as an example, but this can be in Japanese or other languages. In the case of Japanese, "Change Variation" would be replaced with "Variation Change". The variation change button 1170 is operated to instruct the automatic generation of variations of the template displayed in the template editing area 1022.

[0107] In S907, the poster editing application 108 determines whether the variation change button 1170 has been operated. If the variation change button 1170 has been operated by the poster editor Ub, the poster editing application 108 proceeds to S908 and executes the variation generation process. The variation generation process will be described later. If the variation change button 1170 has not been operated in S907, the process proceeds to S909.

[0108] In S909, the poster editing application 108 determines whether or not content within the template has been selected. The poster editing application 108 determines that content within the template has been selected if the image setting object 1061 within the template 1052 displayed in the template editing area 1022 has been instructed and operated by the poster editor Ub via the input unit 307. If content within the template has been selected, the process proceeds to S910 and the content change process is executed. Otherwise, the process proceeds to S911.

[0109] Figure 12 is a flowchart showing the flow of the content change process executed in S910. This flowchart is initiated when it is determined in S909 that content within the template has been selected. The process shown in this flowchart is written in a program stored in the ROM 302 or HDD 304 of client 102. The program is called by the CPU 301 of client 102, loaded into RAM 303, and executed by the CPU 301. In the following description, the symbol "S" represents a step.

[0110] In S1201, the poster editing application 108 displays a dialog box. The dialog box is a screen that accepts the action to be performed on the content selected in S909.

[0111] Figure 13 illustrates the screen transitions of the editing screen during the content modification process. Figure 13(a) shows the editing screen 1310 when the coffee cup image 1181 included in template 1052 is selected and a dialog box 1300 is displayed as a pop-up. The dialog box 1300 includes an automatic generation button 1301 that requests image generation from the image generation AI 480 and an image selection button 1302 that selects an image from a local folder.

[0112] If the poster editor Ub selects the auto-generate button 1301 in dialog 1300 in S1202, proceed to S1204. If the auto-generate button 1301 is not selected, and the image selection button 1302 is selected in S1203, proceed to S1209. If neither the auto-generate button 1301 nor the image selection button 1302 is selected, return to S1202 and wait for a selection.

[0113] In S1204, the poster editing application 108 displays an input area 1321 for additional prompts in the content editing area 1023.

[0114] Figure 13(b) shows an example of the editing screen 1320 with the automatic generation button 1301 selected. The input area 1321 is an input area that accepts new prompts (additional prompts) for the selected image 1181. The display area 1323 is an area that displays the image 1324 generated by the image generation AI 480.

[0115] In S1205, the poster editing application 108 accepts input for an additional prompt in the input area 1321. Text representing the image desired by the poster editor Ub is entered as the additional prompt. In the example in Figure 13(b), the text 1322 "An image of a coffee with a spoon" is entered by the poster editor Ub as the additional prompt.

[0116] In S1206, the content editing unit 424 of the poster editing application 108 sends the input prompt to the image generation AI 480 of the server 101 along with an image generation request. In response to this request, the image generation AI 480 generates, for example, an image 1324 of a coffee cup with a spoon.

[0117] In S1207, the content editing unit 424 of the poster editing application 108 acquires the image 1324 generated by the image generation AI 480.

[0118] In S1208, the content editing unit 424 of the poster editing application 108 displays the acquired image 1324 in the display area 1323. The content editing unit 424 also changes the image in the image setting object 1061 of template 1052 displayed in the template editing area 1022 to the acquired image 1324.

[0119] As mentioned above, the image generation AI 480 has the context 711, which corresponds to template 1052, loaded into it. By loading context 711, the image generation AI 480 generates an image that matches the design of the existing template 1052. In this example, the context 711 associated with template 1052 contains the text "illustrative design," so the image generation AI 480 generates an illustration-style image even without the poster editor Ub explicitly prompting for "illustrative."

[0120] If the image generation AI 480 does not load the context 711, it may generate an image unsuitable for the template 1052, such as a realistic image of a coffee cup. In such cases, the user has to repeatedly search for and input alternative prompts and try to generate an image with the image generation AI 480 until the desired image is generated, which is cumbersome. In this embodiment, by loading the context corresponding to the template into the image generation AI 480, the image generation AI 480 can generate an image that matches the design of the target template. Although Figure 13 shows an example of replacing an image included in the template with another image, this disclosure is also applicable when adding a new image setting object to the template 1052 and adding a newly generated image.

[0121] Next, we will explain the process that occurs when the image selection button 1302 in the dialog box 1300 shown in Figure 13(a) is selected.

[0122] In S1209, the content editing unit 424 of the poster editing application 108 displays a screen for selecting an image from a local folder on the editing screen 1310. In S1210, the poster editor Ub selects the desired image from the local memory of client 102.

[0123] In S1211, the content editing unit 424 of the poster editing application 108 places the image selected in S1210 into the image setting object 1061 in the template 1052. Then, proceeding to S1208, the content editing unit 424 displays the template on which the selected image has been placed and terminates the process shown in the flowchart. Note that the image selection range in the process from S1209 to S1210 is not limited to local storage, but may also include images on a server or in the cloud. Once this flowchart is complete, proceed to S911 in Figure 9.

[0124] Since editing a poster can be time-consuming, poster editor Ub may not finish editing the poster after launching the poster editing application 108 just once. For such cases, the poster editing application 108 has a function to save the poster data being edited. This saving function also includes a context memory function. With the saving function, for example, when poster editor Ub operates the "Save" button (not shown) on the editing screen, the poster editing application 108 saves the poster data in its editing state to a predetermined storage area such as the HDD 304 of client 102, cloud storage, or the HDD of server 101. By saving the poster data being edited in this way, the next time the poster editing application 108 is launched, it can read the poster data being edited.

[0125] The additional prompt entered in the prompt input area 1321 of the editing screen 1320 shown in Figure 13(b) represents the poster data being edited. Therefore, when saving the poster data being edited, it is desirable to include this additional prompt in the context associated with the original template to create a new context. For this reason, the poster editing application 108 of this disclosure further has a context update function.

[0126] If saving the poster data is instructed in S911, the process proceeds to S912. For example, if the "Save" button on the editing screen is operated by the poster editor Ub, the poster editing application 108 determines that saving the poster data has been instructed. If the "Save" button is not operated, the process returns to S907 and waits for an operation in dialog 1300.

[0127] In S912, the poster editing application 108 generates and updates a context to include any additional prompts entered during poster editing in the context associated with the template being edited. As illustrated in the example in Figure 13(b), the poster editing application 108 generates (acquires) a new context that includes the context 711 corresponding to the template 1052 being edited, and the additional prompt 1322 entered during content editing, which is "An image of a coffee with a spoon".

[0128] In S913, the poster editing application 108 stores the poster data being edited in association with the newly generated (acquired) context when saving the poster data. The save location is a predetermined storage area such as the HDD 304 of the client 102, cloud storage, or the HDD of the server 101. The poster data being edited is data to be edited that has had content such as images and text added or changed, or the arrangement of content changed, to be added to the data of the template being edited. Like the template data, it is represented as HTML text.

[0129] The method for updating the context in S912 is the same as the method for obtaining the context by the context acquisition unit 412 of the template creation application 107. That is, the poster editing application 108 sends a context generation request to the text generation AI 470 of the server 101. When requesting context generation, the poster editing application 108 requests the text generation AI 470 of the server 101 to summarize the original context 711 associated with the template to be edited and the additional prompts 1322 entered during poster editing. If the additional prompts 1322 were entered in an interactive format, the data held in RAM 303 as the interactive history, along with the original context, is requested to be summarized by the text generation AI 470 of the server 101. The interactive history includes at least one of the following: additional prompts entered by the user (first prompt), prompts converted into a format suitable for input by the generation AI (second prompt), and poster data being edited (HTML format text).

[0130] When the text generation AI 470 receives a request from the poster editing application 108, it summarizes the items to be summarized that were received along with the request and generates a context. The text generation AI 470 sends the generated context to the poster editing application 108 (client 102). In this way, the poster editing application 108 obtains the context generated by the text generation AI 470 and stores it as an updated context, associated with the poster data being edited.

[0131] The updated context will be loaded along with the poster data the next time poster editor Ub launches the poster editing application 108 and loads the poster data. The poster editing application 108 will then load the loaded updated context into the template generation AI 400. The poster editing application 108 may also display the acquired context in the prompt input area of ​​the editing screen.

[0132] Next, we will explain the variation generation process executed in S908 in Figure 9. Figure 14 is a flowchart showing the flow of the variation generation process. This flowchart is started when it is determined in S907 that the variation change button 1170 has been operated. The process shown in this flowchart is written in a program stored in the ROM 302 or HDD 304 of the client 102. The program is called by the CPU 301 of the client 102, loaded into RAM 303, and executed by the CPU 301. In the following explanation, the symbol "S" represents a step. It is assumed that, before this process starts, the template 1052 is displayed in the template editing area 1022 through the processes of S901 to S906, and the context 711 associated with the template 1052 is input to the generation AI.

[0133] In S1401, the template editing unit 425 of the poster editing application 108 generates a prompt instructing a variation change, such as "Change the template variation".

[0134] In S1402, the template editing unit 425 sends a prompt to the template generation AI 400 on the server 101 instructing it to modify the variation generated in S1401.

[0135] In S1403, the template editing unit 425 may generate a prompt instructing the server 101 to change an image in the template and request the image generation AI 480 to generate an image. As mentioned above, the template generation AI 400 and the image generation AI 480 have the context 711 associated with the selected template 1052 loaded.

[0136] When the template generation unit 401 of the template generation AI 400 receives a prompt from the poster editing application 108 to change the variation, it generates a different template with a different variation for the target template 1052. Also, if image generation is requested, when the image generation unit 403 of the image generation AI 480 receives a prompt from the poster editing application 108 to change the image, it generates a different image based on the context and returns it to the requester.

[0137] In S1404, the template generation unit 401 acquires the template and image generated by the generation AI.

[0138] In S1405, the template generation unit 401 creates template variations from the acquired template and image. In S1406, the template generation unit 401 displays variations of the template created in S1405.

[0139] Figure 15 illustrates the process of changing a template variation. Figure 15(a) shows the editing screen 1110 before the variation change, and Figure 15(b) shows the editing screen 1510 after the variation change. In the example in Figure 15, the original template 1052 is changed to the modified template 1500, as shown in Figure 15(b), by operating the variation change button 1170. In the modified template 1500, the text is changed from "Cafe Open" to another text 1502 such as "Grand Open!!", and is displayed within the text setting object 1501. The position of the text setting object 1501 is also changed from the bottom to the top of the template. The image is also changed from an image of a coffee cup without a spoon 1181 to an image of a coffee cup with a spoon 1504, and is displayed within the image setting object 1503. The position of the image setting object 1503 is also changed from the top to the bottom of the template. The image of the coffee cup with a spoon 1504 is an image created by the image generation AI 480. Since the image generation AI 480 also has the context 711, which is previously associated with template 1052, loaded, an illustration-style image similar to image 1181 included in the template 1052 before the change is generated. Generally, since generation AIs generate images probabilistically, the same image is not generated every time. Therefore, by changing the variation, a different image is generated while maintaining the illustration-style taste. In the example in Figure 15, when the template is changed, the image included in the template is also created and changed by the image generation AI 480, but this disclosure is not limited to that. With regard to the image, it is also possible to keep the image included in the original template and only change the other text elements (text content, color, font, size, etc.) in the variation.

[0140] As described above, the template creation application 107 and poster editing application 108 executed by the information processing device of this disclosure allow for the creation of templates using a generation AI and the editing of poster data using those templates. In this process, a context of prompts entered by the user (template creator Ua) is generated and stored in association with the template. Furthermore, when editing a poster or other creative work using that template, the information processing device acquires the context associated with the template when it acquires the template, and loads it into the generation AI used for editing. Therefore, when performing editing on that template, such as generating variations or generating content included in the template, it becomes possible to have the generation AI generate and acquire variations and content that match the original template. Thus, the workload on the user to obtain the desired creative work when editing a poster or other creative work can be reduced.

[0141] The editing features in the poster editing app are not limited to the generation of template variations and content included in templates by the AI, as illustrated by the examples. Users may also be allowed to manually modify images and text, and manually change the placement of such content.

[0142] <Variation> In the embodiments described above, an example was shown where the template creation application 107 and the poster editing application 108 were separate applications. However, the present disclosure may also be configured as a single application that includes the functions of the template creation application 107 and the poster editing application 108. That is, the application of the present disclosure includes a prompt reception unit 410, a template acquisition unit 411, a context acquisition unit 412, a storage unit 413, an acquisition unit 421, a context input unit 422, an editing unit 424, and a context update unit 426. Each of these functional units works in cooperation with the generation AI of the backend application 105 to create templates and edit posters using the created templates. The details of each functional unit are the same as in the embodiments described above.

[0143] Preferred embodiments of this disclosure have been described above with reference to the attached drawings, but this disclosure is not limited to such examples. The example prompt inputs, AI-generated responses, screen examples, and processing procedures provided are examples only and can be modified as appropriate without departing from the spirit of this disclosure. Furthermore, it is clear that those skilled in the art can conceive of various modifications or alterations within the scope of the disclosed technical idea, and these will naturally also fall within the technical scope of this disclosure.

[0144] <Other Embodiments> This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0145] The above-described embodiments include the following configurations.

[0146] (Composition 1) A means for obtaining the context of the prompt used to generate the template, along with the template itself, An input means for inputting the context acquired by the acquisition means into a generating AI (Artificial Intelligence), An editing means that uses the generating AI into which the aforementioned context has been input to edit the production data using the template acquired by the acquisition means, An information processing device characterized by comprising:

[0147] (Configuration 2) The aforementioned generating AI includes a first generating AI that has been trained for template generation, The information processing device according to configuration 1, characterized in that the input means inputs the context to the first generating AI.

[0148] (Composition 3) The aforementioned generating AI includes a second generating AI that has been trained for image generation, The information processing device according to configuration 1 or configuration 2, characterized in that the input means inputs the context to the second generating AI.

[0149] (Composition 4) The information processing device according to any one of configurations 1 to 3, characterized in that the editing means accepts input of an additional prompt by the user, inputs the additional prompt to the generating AI, and outputs a modified template including the processing result by the generating AI as edited production data.

[0150] (Composition 5) The information processing apparatus according to configuration 4, further comprising an update means for updating the context acquired by the acquisition means into a new context including the additional prompt, and storing it in association with the production data.

[0151] (Composition 6) The information processing apparatus according to any one of configurations 1 to 5, characterized in that the editing means performs a process to change the content included in the template.

[0152] (Composition 7) The information processing apparatus according to any one of configurations 1 to 6, characterized in that the editing means performs a process to generate variations of the template.

[0153] (Composition 8) The information processing device according to any one of configurations 1 to 7, characterized in that the context to be summarized includes at least one of the following: a first prompt which is a prompt entered by the user into the generating AI; a second prompt which is the first prompt converted into a format suitable for input to the generating AI; and the generating AI's response in the dialogue between the user and the generating AI.

[0154] (Composition 9) The information processing apparatus according to any one of configurations 1 to 8, characterized in that the context is text data.

[0155] (Composition 10) The information processing device according to any one of configurations 1 to 9, characterized in that the template is text data in HTML format.

[0156] (Composition 11) The information processing apparatus according to any one of configurations 1 to 10, characterized in that, when the generating AI can receive an image as a prompt, the input means inputs an image included in the template or an image indicating the template to the generating AI, along with the context acquired by the acquisition means.

[0157] (Composition 12) The information processing device according to any one of configurations 1 to 11, characterized in that the model of the generated AI is updated based on the context input by the input means.

[0158] (Composition 13) A means of receiving prompt input from the user, A first acquisition means that acquires a template generated by a generating AI based on the prompt received by the reception means, A second acquisition means for acquiring the context of the prompt used to generate the template, A storage means that stores the context acquired by the second acquisition means in association with the template acquired by the first acquisition means, An information processing device characterized by comprising:

[0159] (Composition 14) The information processing apparatus according to configuration 13, characterized in that the receiving means accepts input of the prompt in a dialogue format between the user and the generating AI.

[0160] (Composition 15) The information processing device according to configuration 13 or configuration 14, characterized in that the context to be summarized includes a first prompt which is the prompt received by the receiving means, a second prompt which is the first prompt converted into a format suitable for input to the generating AI, and the response of the generating AI in the dialogue between the user and the generating AI.

[0161] (Composition 16) The information processing apparatus according to any one of configurations 13 to 15, characterized in that the second acquisition means acquires the processing result by the third generation AI, which has been trained for context generation, as the context.

[0162] (Composition 17) The information processing device according to configuration 16, characterized in that the model of the third generating AI is updated by the prompt input to the third generating AI.

[0163] (Composition 18) The information processing apparatus according to any one of configurations 13 to 17, characterized in that the second acquisition means acquires the context as text data.

[0164] (Composition 19) The information processing apparatus according to any one of configurations 13 to 18, characterized in that the context acquired by the second acquisition means includes the template data acquired by the first acquisition means.

[0165] (Composition 20) The information processing apparatus according to any one of configurations 13 to 19, characterized in that the storage means stores the context in association with the category to which the template acquired by the first acquisition means belongs.

[0166] (Composition 21) The information processing apparatus according to any one of configurations 13 to 20, characterized in that the first acquisition means acquires the template as text data in HTML format.

[0167] (Composition 22) A means of receiving prompt input from the user, A first acquisition means that acquires a template generated by a generating AI based on the prompt received by the reception means, A second acquisition means for acquiring the context of the prompt used to generate the template, A storage means that stores the context acquired by the second acquisition means in association with the template acquired by the first acquisition means, A third acquisition means for acquiring the context associated with the template, together with the template, An input means for inputting the context acquired by the third acquisition means into the generating AI, An editing means that uses the generating AI into which the aforementioned context has been input to edit the production data using the template acquired by the third acquisition means, An information processing device characterized by comprising:

[0168] (Composition 23) An information processing method performed by an information processing device, A step of obtaining the prompt context used to generate the template, along with the template, The steps include inputting the acquired context into the generating AI, The steps include: using the generating AI to input the aforementioned context, editing the production data using the acquired template; An information processing method characterized by including

[0169] (Composition 24) An information processing method performed by an information processing device, Steps to accept prompt input from the user, The steps include obtaining a template generated by the AI ​​based on the received prompt, A step of obtaining the context of the prompt used to generate the template, The steps include storing the acquired context in association with the acquired template, An information processing method characterized by including

[0170] (Composition 25) An information processing method performed by an information processing device, Steps to accept prompt input from the user, The steps include obtaining a template generated by the AI ​​based on the received prompt, The steps include obtaining the context of the prompt used to generate the template, The steps include storing the acquired context in association with the acquired template, A step of obtaining the context of the prompt used to generate the template, along with the template, The steps include inputting the acquired context into the generating AI, The steps include: using the generating AI to input the aforementioned context, editing the production data using the acquired template; An information processing method characterized by including

[0171] (Composition 26) Computers, A means for obtaining the context of the prompt used to generate the template, along with the template itself, An input means for inputting the context acquired by the acquisition means into the generating AI, An editing means that uses the generating AI into which the aforementioned context has been input to edit the production data using the template acquired by the acquisition means, A program designed to function as such.

[0172] (Composition 27) Computers, A means of receiving prompt input from the user, A first acquisition means that acquires a template generated by a generating AI based on the prompt received by the reception means, A second acquisition means for acquiring the context of the prompt used to generate the template, A storage means that stores the context acquired by the second acquisition means in association with the template acquired by the first acquisition means, A program designed to function as such.

[0173] (Composition 28) Computers, A means of receiving prompt input from the user, A first acquisition means that acquires a template generated by a generating AI based on the prompt received by the reception means, A second acquisition means for acquiring the context of the prompt used to generate the template, A storage means that stores the context acquired by the second acquisition means in association with the template acquired by the first acquisition means, A third acquisition means for acquiring the context associated with the template, together with the template, An input means for inputting the context acquired by the third acquisition means into the generating AI, An editing means that uses the generating AI into which the aforementioned context has been input to edit the production data using the template acquired by the third acquisition means, A program designed to function as such.

Claims

1. A means for obtaining the context of the prompt used to generate the template, along with the template itself, An input means for inputting the context acquired by the acquisition means into a generating AI (Artificial Intelligence), An editing means that uses the generating AI into which the aforementioned context has been input to edit the production data using the template acquired by the acquisition means, An information processing device characterized by comprising:

2. The aforementioned generation AI includes a first generation AI that has been trained for template generation, The information processing device according to claim 1, characterized in that the input means inputs the context to the first generating AI.

3. The aforementioned generation AI includes a second generation AI that has been trained for image generation, The information processing device according to claim 1, characterized in that the input means inputs the context to the second generating AI.

4. The information processing apparatus according to claim 1, characterized in that the editing means accepts input of an additional prompt from the user, inputs the additional prompt to the generating AI, and outputs a modified template including the processing result by the generating AI as edited production data.

5. The information processing apparatus according to claim 4, further comprising an update means for updating the context acquired by the acquisition means into a new context including the additional prompt and storing it in association with the production data.

6. The information processing apparatus according to claim 1, wherein the editing means performs a process to change the content included in the template.

7. The information processing apparatus according to claim 1, wherein the editing means performs a process to generate variations of the template.

8. The information processing apparatus according to claim 1, characterized in that the context to be summarized includes at least one of the following: a first prompt which is a prompt entered by the user into the generating AI; a second prompt which is the first prompt converted into a format suitable for input to the generating AI; and the response of the generating AI in the dialogue between the user and the generating AI.

9. The information processing apparatus according to claim 1, characterized in that the context is text data.

10. The information processing apparatus according to claim 1, characterized in that the template is text data in HTML format.

11. The information processing apparatus according to claim 1, wherein, if the generating AI can receive an image as a prompt, the input means inputs an image included in the template or an image representing the template to the generating AI, along with the context acquired by the acquisition means.

12. The information processing apparatus according to claim 1, characterized in that the model of the generated AI is updated based on the context input by the input means.

13. A means of receiving prompt input from the user, A first acquisition means that acquires a template generated by a generation AI based on the prompt received by the reception means, A second acquisition means for acquiring the context of the prompt used to generate the template, A storage means that stores the context acquired by the second acquisition means in association with the template acquired by the first acquisition means, An information processing device characterized by comprising:

14. The information processing apparatus according to claim 13, characterized in that the receiving means receives input of the prompt in a dialogue format between the user and the generating AI.

15. The information processing apparatus according to claim 13, characterized in that the context to be summarized includes a first prompt which is the prompt received by the receiving means, a second prompt which is the first prompt converted into a format suitable for input to the generating AI, and the response of the generating AI in the dialogue between the user and the generating AI.

16. The information processing apparatus according to claim 13, characterized in that the second acquisition means acquires the processing result by the third generating AI, which has been trained for context generation, as the context.

17. The information processing device according to claim 16, characterized in that the model of the third generating AI is updated by the prompt input to the third generating AI.

18. The information processing apparatus according to claim 13, characterized in that the second acquisition means acquires the context as text data.

19. The information processing apparatus according to claim 13, characterized in that the context acquired by the second acquisition means includes the template data acquired by the first acquisition means.

20. The information processing apparatus according to claim 13, characterized in that the storage means stores the context in association with the category to which the template acquired by the first acquisition means belongs.

21. The information processing apparatus according to claim 13, characterized in that the first acquisition means acquires the template as text data in HTML format.

22. A means of receiving prompt input from the user, A first acquisition means that acquires a template generated by a generation AI based on the prompt received by the reception means, A second acquisition means for acquiring the context of the prompt used to generate the template, A storage means that stores the context acquired by the second acquisition means in association with the template acquired by the first acquisition means, A third acquisition means for acquiring the context associated with the template, together with the template, An input means for inputting the context acquired by the third acquisition means into the generating AI, An editing means that uses the generated AI into which the aforementioned context has been input to edit the production data using the template acquired by the third acquisition means, An information processing device characterized by comprising:

23. An information processing method performed by an information processing device, A step of obtaining the prompt context used to generate the template, along with the template, The steps include inputting the acquired context into the generating AI, The steps include: editing the production data using the acquired template with the generated AI into which the aforementioned context has been input; An information processing method characterized by including

24. An information processing method performed by an information processing device, Steps to accept prompt input from the user, The steps include obtaining a template generated by the AI ​​based on the received prompt, A step of obtaining the context of the prompt used to generate the template, The steps include storing the acquired context in association with the acquired template, An information processing method characterized by including

25. An information processing method performed by an information processing device, Steps to accept prompt input from the user, The steps include obtaining a template generated by the AI ​​based on the received prompt, The steps include obtaining the context of the prompt used to generate the template, The steps include storing the acquired context in association with the acquired template, A step of obtaining the context of the prompt used to generate the template, along with the template, The steps include inputting the acquired context into the generating AI, The steps include: editing the production data using the acquired template with the generated AI into which the aforementioned context has been input; An information processing method characterized by including

26. Computers, A means for obtaining the context of the prompt used to generate the template, along with the template itself, An input means for inputting the context acquired by the acquisition means into the generating AI, An editing means that uses the generating AI into which the aforementioned context has been input to edit the production data using the template acquired by the acquisition means, A program designed to function as such.

27. Computers, A means of receiving prompt input from the user, A first acquisition means that acquires a template generated by a generation AI based on the prompt received by the reception means, A second acquisition means for acquiring the context of the prompt used to generate the template, A storage means that stores the context acquired by the second acquisition means in association with the template acquired by the first acquisition means, A program designed to function as such.

28. Computers, A means of receiving prompt input from the user, A first acquisition means that acquires a template generated by a generation AI based on the prompt received by the reception means, A second acquisition means for acquiring the context of the prompt used to generate the template, A storage means that stores the context acquired by the second acquisition means in association with the template acquired by the first acquisition means, A third acquisition means for acquiring the context associated with the template, together with the template, An input means for inputting the context acquired by the third acquisition means into the generating AI, An editing means that uses the generated AI into which the aforementioned context has been input to edit the production data using the template acquired by the third acquisition means, A program designed to function as such.

Images