Information processing device, information processing method, and program

Abstract

To facilitate setting of an appropriate prompt for obtaining a content of a desired style with a generative AI.SOLUTION: An information processing device obtains both a base image that is a base of a content desired to be generated by a generative AI and a style image that represents a style of the content; extracts attribute information indicating the style represented by the style image, from the obtained style image; and sets in the generative AI a prompt for making the generative AI generate the content based on the base image, based on the extracted attribute information.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to prompt settings for image generation AI. [Background technology]

[0002] A service is provided that supports the production of posters, flyers, etc., so that anyone can easily produce a finished product of a certain quality by adding any text or image to a desired template selected from a variety of pre-prepared templates.

[0003] There are cases where the image content included in a selected template is edited while maintaining the layout of the template. For example, if you want to change a poster for an event in the hot season to a poster for an event in the cold season, if the people in the poster are wearing light clothing, you will need to change them to thicker clothing suitable for the cold season. If there are multiple people in the poster, you will need to change the clothing of all of them.

[0004] Recently, it has become possible to generate necessary content using generative AI (Artificial Intelligence) technology. With generative AI technology, when a user inputs images or text as input prompts into a generative model, it can generate characters, images, videos, etc. that are highly likely to match the "context" expressed by the input prompt. Using this technology, users can easily change multiple image contents included in a template. However, it is necessary to edit the multiple image contents according to the same context.

[0005] Patent document 1 discloses a technology that extracts information indicating the type of image from the property information of an object contained in a template, and creates a prompt to generate an image that matches the template based on that information. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-37557 [Non-patent literature]

[0007] [Non-Patent Document 1] “Stable Diffusion”, [online], [Retrieved February 8, 2024], Internet<URL:https: / / stability.ai / > [Non-patent document 2] “ChatGPT (registered trademark)”, [online], [searched February 8, 2024], Internet<URL:https: / / chat.openai.com / > [Non-patent document 3] “GPT-3”, [online], [Retrieved February 8, 2024], Internet<URL:https: / / github.com / openai / gpt-3> [Non-patent document 4] “Show and Tell”, [online], [Retrieved February 8, 2024], Internet<URL:https: / / arxiv.org / abs / 1411.4555> [Non-Patent Document 5] “Neural Style Transfer”, [online], [Retrieved February 8, 2024], Internet<URL:https: / / arxiv.org / abs / 1508.06576> Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the technology of Patent Document 1, it is necessary to create property information corresponding to each of the various templates and check the property information, making it difficult to set an appropriate prompt. [Means for solving the problem]

[0009] An information processing device according to one embodiment of the present disclosure is an information processing device for causing a generation AI to generate content, and is characterized by having an acquisition means for acquiring a base image that forms the basis of the content that the generation AI is to generate and a style image that represents the style of the content, an extraction means for extracting attribute information that indicates the style represented by the style image from the style image acquired by the acquisition means, and a setting means for setting a prompt to the generation AI for the generation AI to generate the content based on the base image based on the attribute information extracted by the extraction means. [Effects of the Invention]

[0010] According to the technology of the present disclosure, it is possible to easily set an appropriate prompt to obtain content with a desired style. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image output device. [Figure 3] FIG. 2 is a diagram illustrating an example of the hardware configuration of a client PC and a server. [Figure 4] FIG. 2 is a functional block diagram of the information processing system. [Figure 5] FIG. 10 is a diagram illustrating an example of image generation processing. [Figure 6] FIG. 10 is a diagram illustrating an example of a layout data DB. [Figure 7] FIG. 10 is a diagram showing an example of a layout data editing screen. [Figure 8] 10 is a flowchart showing the flow of an image generation process. [Figure 9] FIG. 10 is a diagram showing an example of a layout data editing screen. [Figure 10] FIG. 2 is a functional block diagram of the information processing system. [Figure 11]FIG. 10 is a diagram showing an example of a layout data editing screen. [Figure 12] FIG. 10 is a diagram illustrating an example of a layout data DB. [Figure 13] 10 is a flowchart showing the flow of an image generation process. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the technology of the present disclosure will be described in detail with reference to the drawings. Note that the following embodiments do not limit the technology of the present disclosure according to the claims, and not all combinations of features described in the embodiments are necessarily essential to the solution of the technology of the present disclosure. In the accompanying drawings, the same reference numerals are used to designate identical or similar components, and redundant explanations will be omitted. Each process (step) in the flowchart is denoted by an "S" at the beginning.

[0013] <<Embodiment 1>> An information processing system according to this embodiment will be described. The information processing system according to this embodiment is a printing system that involves editing layout data for an image output device. In the printing system, an externally connected client PC edits the layout data and transmits a print job to the image output device. When creating a print job, print settings are edited as needed on a screen displayed on the display of a display device included in the client PC.

[0014] (Configuration of information processing system) 1 is a diagram showing an example of the configuration of an information processing system according to this embodiment. The information processing system according to this embodiment includes an image output device A 101, an image output device B 102, a client PC 103, and a server 105, and each device is connected to each other via a network 104 such as Ethernet so that data can be exchanged between them.

[0015] A layout data creation application is installed on the client PC 103, and by executing the layout data creation application, the user edits layout data such as posters and flyers. The client PC 103 requests the server 105 to perform some editing and data processing related to the layout data and rendering processing. Furthermore, the client PC 103 generates a print job by adding print settings to the edited layout data, and transmits the generated print job to the image output devices 101 and 102. In this embodiment, the number of image output devices is two, but this is not limited to this, and the number may be one or three or more. The number of client PCs and servers is also one, but two or more may be.

[0016] As an example of printing execution, this embodiment will explain a system in which a print job is sent from a printing application installed on a client PC to an image output device A101 via a printer driver. For example, a printing application and a printer driver are installed on a client PC 103. The printing application can obtain device information of the associated image output device A101 from the printer driver, as well as printing parameters such as paper type, paper size, and print quality, and can edit print settings from the obtained printing parameters.

[0017] A print job is created based on the print settings and the layout data image that has been rendered by the server 105, and the print job is sent to the image output device via the print driver spool, thereby executing the print process. The image output device then executes printing based on the print settings of the received print job. The image output device also stores configuration information related to the ink and paper it handles, as well as status information such as idle state and print errors, as device information. Furthermore, if printing cannot be executed normally due to a problem with the image output device, such as insufficient paper or empty ink, or an error in the print settings, a warning message is displayed on the device panel to inform the user of the reason why printing cannot be executed normally.

[0018] (Hardware configuration of image output device) FIG. 2 is a diagram illustrating an example of the hardware configuration of the image output device A101. The image output device B102 also has a similar hardware configuration to the image output device A101, and therefore a description of the hardware configuration of the image output device B102 will be omitted. The image output device A101 is controlled by a CPU 201. The CPU 201 operates based on a control program stored in a program ROM of the ROM 202 or a control program stored in an external memory 209. The CPU 201 outputs an image signal as output information to a printing unit (printer engine) 208 connected to a printing unit I / F 206 via a system bus 204. The CPU 201 is capable of communication with the client PC 103 via an input unit 205 and can notify the client PC 103 of information stored in the image output device A101. The CPU 201 can also receive output data to be output to the printing unit 208 via the input unit 205. The RAM 203 functions as the main memory, work area, etc. of the CPU 201, and is configured so that its memory capacity can be expanded by an optional RAM connected to an expansion port (not shown). The RAM 203 is used as an output information development area, an environmental data storage area, non-volatile memory, etc. The external memory 209 is configured as a hard disk (HDD), an IC card, etc., and access to it is controlled by a memory controller 207. The external memory 209 is connected as an option and stores font data, emulation programs, form data, ink used, information about the type and size of paper fed, main unit status information, etc. The operation unit 210 is equipped with a panel and is configured so that various information can be displayed.

[0019] (Client PC and server hardware configuration) FIG. 3 is a block diagram showing an example of the hardware configuration of the client PC 103 and the server 105. The client PC 103 and the server 105 are, for example, information processing devices such as personal computers (PCs). The client PC 103 and the server 105 share a common hardware configuration, including a computer interior 308. The computer interior 308 includes a CPU 301, a ROM 302, a RAM 303, a keyboard controller 305, a display controller 306, and a disk controller 307. The CPU 301 loads various programs, such as a control program, a system program, and an application program, from an external memory 311 to the RAM 303 via the disk controller 307. The CPU 301 then executes the various programs loaded into the RAM 303 to perform various data processing operations and control the display 310. The CPU 301 may also load the control program, etc., from the ROM 302. The CPU 301 may be a dedicated circuit such as an ASIC. The CPU 301 and the dedicated circuit are examples of hardware circuits and hardware processors.

[0020] A disk controller 307 controls access to an external memory 311 such as a HD, CD-ROM, DVD-ROM, or USB. A RAM 303 is configured so that its capacity can be expanded using an optional RAM (not shown) or the like, and is used primarily as a work area for the CPU 301. A keyboard controller 305 controls key input from a keyboard 309 or a pointing device (not shown).

[0021] The display controller 306 controls the display on the display 310. In this embodiment, unless otherwise specified, the CPU 301 controls each unit connected to the main bus 304 via the main bus 304. Of course, in the server 105, devices that are not necessarily required, such as the display 310, do not necessarily have to be included in the configuration.

[0022] (Functional configuration of information processing system) 4 is a block diagram showing an example of the functional configuration of the information processing system of this embodiment. In the information processing system of this embodiment, the output target is an image output device A 101. First, the functional configurations within the client PC 103 and server 105 will be described.

[0023] (Client PC functional configuration) The client PC 103 includes a layout data DB 411 , a layout data editing unit 412 , an image generation request unit 413 , a style image input unit 414 , a content image input unit 415 , and a print job transmission unit 416 .

[0024] The layout data DB 411 stores layout data, which will be described in detail later with reference to FIG.

[0025] The layout data editing unit 412 adds or deletes content such as text and images to be placed on posters and flyers, and adjusts the layout of each content. When processing content such as cutting out or filling in, the layout data editing unit 412 requests the data content editing unit 421 of the server 105 to perform the processing. The layout data is saved as a cache in the layout data DB 411 of the client PC 103. Alternatively, the layout data is saved in the layout data DB 422 of the server 105 for each client PC 103 (or for each account, if a user account exists).

[0026] The image generation request unit 413 requests the image generation unit 423 of the server 105 to generate an image based on the image information to be used for image generation set in the style image input unit 414 and the image information set in the content image input unit 415.

[0027] The style image input unit 414 sets an ID indicating a style image, which is image information used for image generation. The content image input unit 415 sets an ID indicating the content to be converted, which is a base image that forms the basis of the content to be generated by the generation AI.

[0028] The print job sending unit 416 creates a print job and sends the created print job to the image output device A 101. When creating the print job, the print job sending unit 416 requests a preview image generation unit 425 of the server 105 to preview the layout data, and requests a print image generation unit 426 to generate a print image.

[0029] (Functional configuration of server 105) The server 105 includes a data content editing unit 421 , a layout data DB 422 , an image generating unit 423 , a generative model 424 , a preview image generating unit 425 , a print image generating unit 426 , a prompt generating unit 427 , and an attribute information generating unit 428 .

[0030] The data content editing unit 421 edits the content by performing processing such as cutting out and filling in the content. The layout data DB 422 is synchronized with the layout data DB 411 and stores the same layout data as the layout data stored in the layout data DB 411. Details of the layout data will be described later with reference to FIG. 5.

[0031] The image generation unit 423 first acquires attribute information generated by the attribute information generation unit 428 based on the image information set in the style image input unit 414, and generates a prompt by the prompt generation unit 427 based on the acquired attribute information. The image generation unit 423 generates a new image based on the generated prompt using the generation model 424. The generated new image is saved in the layout data DBs 411 and 422 and reflected on the layout data editing screen.

[0032] The image generation unit 423 utilizes generative AI technology, taking an input image and an input prompt as inputs and generating a product using a generative model. Specifically, generative models such as Stable Diffusion (see Non-Patent Document 1), ChatGPT (registered trademark) (see Non-Patent Document 2), and GAN (Generative Adversarial Network), a generative adversarial algorithm, are utilized. FIG. 5 illustrates image generation processing using a generative model. Upon receiving input 501 consisting of an input image 511 and an input prompt 512, the generative model 502 can output, as a product 503, an image 513 that is highly likely to match the "context" represented by the input 501. The relationship between the input value and the "context" is acquired when the generative model 502 learns using a large number of images and text. Furthermore, the combination of input and output for generative AI technology varies depending on the generative model used, and users must choose the appropriate generative AI technology and generative model as needed.

[0033] The generative model 424 is a model used by the image generation unit 423 when generating an image. The generative model 424 can output a different image as a product even when the same input image and input prompt are used by changing the initial values, which are mainly generated from random numbers during image generation. When converting the style of an image, such as "watercolor painting," "abstract painting," or "anime style," the following generative model may be used. That is, a generative model may be used that converts an input image to match the taste of the trained image by training using an image with a specific style, as in the case of Neural Style Transfer disclosed in Non-Patent Document 5, and outputs the converted image.

[0034] A preview image generating unit 425 executes a preview of the layout data. A print image generating unit 426 executes processing for generating a print image. A prompt generating unit 427 generates a prompt based on attribute information. The generation of the prompt will be described in detail later.

[0035] The attribute information generation unit 428 performs image recognition using an image recognition model 429 on the image set in the style image input unit 414, and generates attribute information that represents the content of the image. Details of the generation of the attribute information will be described later. The image recognition model 429 is a model used by the attribute information generation unit 428 when generating attribute information.

[0036] (Functional configuration of image output device A) Next, the functional configuration of the image output device A101 will be described. The image output device 101 has a device information holding unit 431, a print job receiving unit 432, and a print execution unit 433. The device information holding unit 431, the print job receiving unit 432, and the print execution unit 433 are connected to the ROM 202 of the image output device 101. The device information holding unit 431 holds information such as the type and remaining amount of ink installed in the image output device 101, information such as the type and size of registered paper and fed paper, main body status information of the image output device A101, and status information of print jobs. The print job receiving unit 432 receives print jobs sent from the client PC 103. The print execution unit 433 executes print processing for the print job.

[0037] Of course, if the image output device A101 has been determined in advance as the image output device to be used, the following processing may be performed to create layout data suitable for the image output device A101. That is, the device information held by the device information holding unit 431 may be acquired, and the acquired device information may be associated with the layout data saved in the layout data DB 411 or 422 and held on the client PC 103 or server 105 side.

[0038] (Layout data) FIG. 6 is a diagram showing an example of layout data stored in the layout data DBs 411 and 422. A data table 600 shown in FIG. 6 exists for each piece of layout data. The data table 600 includes parameters such as an ID 601, data content 602, content type 603, layout coordinates 604, and setting information 605. Identification information for uniquely identifying content in the layout data is registered in the ID 601. In FIG. 6, six IDs, "ID-A," "ID-B," "ID-C," "ID-D," "ID-E," and "ID-F," are registered. The identification information indicated in the ID 601 is linked to information indicated in each of the items: data content 602, content type 603, layout coordinates 604, and setting information 605.

[0039] The data content 602 field contains values ​​for each content item, such as text and images laid out on the layout data. The content type 603 field contains content type information indicating the type of content, such as text, image, document size, and variable information.

[0040] The layout coordinates 604 field contains coordinates that indicate the position of the content on the layout data, with the top left corner as the base. The setting information 605 field contains attribute values ​​for each piece of content, such as the color and size of the content. The setting information 605 field also contains a style image flag indicating whether it is a style image, and a content image flag indicating whether it is a content image. In this embodiment, style images and content images are defined as follows:

[0041] In this embodiment, it is assumed that an image is generated using a generative model that receives an image and text as input and outputs an image, such as Stable Diffusion disclosed in Non-Patent Document 1. Here, the image input to the generative model is referred to as a content image. Also, the image used to generate the text input to the generative model is referred to as a style image. The style image flag and content image flag each express how an image already placed in the layout data is used during image generation.

[0042] It also stores settings related to the entire layout data, such as document size and data for variable printing, with "overall" stored in data content 602, setting types stored in content type 603, and setting values ​​stored in setting information 605. Of course, each parameter type may be managed in a separate file, or parameter types other than those mentioned above may be included in the layout data.

[0043] (Layout data editing screen) 7 is a diagram showing an example of a layout data editing screen according to this embodiment. The layout data editing screen 700 is a UI screen displayed on the display 310 of the client PC 103, with the image output device A 101 as the output target (print execution target). In FIG. 7, it is assumed that a template 710 in the template list 701 is selected, and that content 741 has been added to the template 710 displayed in the layout editing area 704 by a user operation on the add image button 702. It is assumed that content 714 in the layout editing area 704 is selected in order to set whether it is a style image target or a content image target.

[0044] The layout data editing screen 700 displays a template list 701, an add image button 702, an add text button 703, a layout editing area 704, a print execution button 705, a style conversion button 706, and a generation AI function area 707. The generation AI function area 707 displays a style image target checkbox 708 and a content image target checkbox 709.

[0045] A template list 701 displays a plurality of pre-prepared templates 710, 720, and 730 (three in the illustrated example). The user can view and select a template that most closely resembles the completed image of the layout data from the plurality of templates displayed in template list 701. When a template is selected by a user operation, the selected template is displayed in layout editing area 704. In FIG. 7, of the plurality of templates 710, 720, and 730 displayed in template list 701, template 710 is selected by a user operation, and the selected template 710 is displayed in layout editing area 704. Note that the template information of the templates displayed in template list 701 may be acquired as layout data from layout data DB 400 or 410, or may be acquired from an SNS service or another external cloud service.

[0046] The layout editing area 704 is an area where content included in the displayed template can be edited. That is, in the layout editing area 704, it is possible to perform editing such as position adjustment, clipping, and filling in each of the multiple pieces of content displayed by the layout data editing unit 401 and the data content editing unit 409. In addition, by the user pressing an add image button 702 or add text button 703 (described later), it becomes possible to add content such as images and text to the template displayed in the layout editing area 704.

[0047] The add image button 702 is a button for accepting a user operation to add an image to the template displayed in the layout editing area 704. The add text button 703 is a button for accepting a user operation to add text to the template displayed in the layout editing area 704. When the add image button 702 or the add text button 703 is pressed by a user operation, the desired content is added to the template displayed in the layout editing area 704. That is, when a file dialog is called and a file path is specified, an import process is performed to add the desired content. Of course, additional buttons corresponding to other content types may be placed, and the import source may also be specified as an external cloud service storage or SNS service. Furthermore, content may be added to the template displayed in the layout editing area 704 by dragging and dropping.

[0048] The print execution button 705 is a button for accepting a user operation to print the image displayed in the layout editing area 704. When the print execution button 705 is pressed by the user, the layout data editing unit 401 requests the print job sending unit 405 to create a print job and send the created print job to the image output devices 101 and 102. The print job is a print job for the layout data displayed in the layout editing area 704.

[0049] The style conversion button 706 is a button for accepting a user operation to convert selected content to match a style image included in the template image. In other words, the style conversion button 706 is a button for accepting a user operation to convert the style of selected content to match the style of a style image included in the template image. The selected content is, for example, content (image or text) included in the template image displayed in the layout editing area 704 and added to the template image by a user operation on the add image button 702 or the add text button 703. Since the style conversion button 706 is a button for converting the style, it can also be said to accept a prompt for conversion of newly inputted content by a generation AI with a set prompt. Note that the timing of operating the style conversion button 706 may be such that a prompt for conversion is received each time new content is input, or that conversion is performed when a user instruction is received.

[0050] The generation AI function area 707 is an area that indicates whether the target image is a style image target or a content image target when generating an image using the generation AI function. In the generation AI function area 707, a style image target checkbox 708 and a content image target checkbox 709 are displayed for each of the multiple images displayed in the layout editing area 704.

[0051] The style image target checkbox 708 is an item used to set whether the target image is a style image target. The content image target checkbox 709 is an item used to set whether the target image is a content image target.

[0052] Conversion of a content image to harmonize it with the style of the style image displayed in the layout editing area 704 begins when the style conversion button 706 is pressed by a user operation. The image generation request unit 413 sets a content image specified from the images in the layout editing area 704 in the content image input unit 415 and sets the style image in the style image input unit 414. Then, a new image is generated by converting the style of the specified content image so that it harmonizes with the style of the style image, and the generated new image is presented to the user. That is, the style of the specified content image is converted so that it harmonizes with the style of the style image, and the style-converted content image is presented to the user. When multiple content images are specified, the multiple content images are set in the content image input unit 415. When multiple style images are specified, the specified multiple style images are also set in the style image input unit 414. A style image may be specified by displaying a style image target checkbox associated with each of multiple content images in the layout editing area 704 and accepting selection by a user operation. Alternatively, all images in the layout editing area 704 other than the image set as a content image may be selected as style images.

[0053] Furthermore, as for the method of presenting a new image to the user, when a content image in the layout editing area 704 is designated, the designated content image in the layout editing area 704 may be replaced with the new image. Alternatively, the new image may be added as a separate content, allowing the user to select whether to adopt the designated content image or the new image.

[0054] (Image generation processing) FIG. 8 is a flowchart showing the flow of an example of image generation processing according to this embodiment. FIG. 8 shows the flow of processing in which an image matching an image already arranged on the layout data editing screen 700 is generated by a generation AI. That is, all content included in a template is set in advance as a style image, and the flow of processing in which the style of an added image is converted so as to harmonize with the style of the style image is shown. The flowchart shown in FIG. 8 is realized, for example, by the CPU 301 reading a program stored in the ROM 302 into the RAM 303 and executing it.

[0055] 8 by the image generation request unit 413 at the timing when, for example, the style conversion button 706 on the layout data editing screen 700 is pressed. It is assumed that the template selected in the template list 701 and the desired added content image are displayed in the layout editing area 704 of the layout data editing screen 700, and that the added content image is selected.

[0056] In S801, the image generation request unit 413 acquires a content image specified in the layout editing area 704 and sets an ID indicating the acquired content image in the content image input unit 415. When multiple content images are specified in the layout editing area 704, the image generation request unit 413 sets the ID of each image specified as a content image in the content image input unit 415. For example, assume that content 741 is added by a user operation on the image addition button 702, and the content image target checkbox 709 is selected in the generation AI function area 707 displayed in association with the content 741. In this case, the ID associated with the content 741 is set in the content image input unit 415.

[0057] In S802, the image generation request unit 413 acquires the style image specified in the layout editing area 704 and sets an ID indicating the acquired style image in the style image input unit 414. When multiple style images are specified in the layout editing area 704, the image generation request unit 413 sets the IDs of the images specified as style images in the style image input unit 414. For example, assume that the style image target checkbox 708 is selected in the generation AI function area 707 displayed in association with content 714 in the layout editing area 704. In this case, the ID associated with the content 714 is set in the style image input unit 414.

[0058] In S803, the image generation request unit 413 requests the image generation unit 423 of the server 105 to generate a new image based on the information set in the content image input unit 415 and the information set in the style image input unit 403. That is, the image generation request unit 413 requests the image generation unit 423 to generate a style-converted content image in which the style of the content image is converted so as to harmonize with the style of the style image. For example, the image generation request unit 413 requests the image generation unit 423 to generate an image based on the ID of the content 741 set in the content image input unit 415 and the ID of the content 714 set in the style image input unit 414.

[0059] In S804, the attribute information generation unit 428 acquires attribute information representing the content of the image from the image set in the style image input unit 414 using image recognition technology. The image recognition technology may use a model trained to classify specific elements, such as season, event, or style. A generative model, such as Show and Tell disclosed in Non-Patent Document 4, may also be used, which receives an image as input and generates explanatory text. The attribute information may include, for example, information indicating a season, such as "spring" or "winter," information indicating an event, such as "Christmas" or "Halloween," or information indicating a style, such as "watercolor painting," "abstract painting," or "anime style." For example, the attribute information generation unit 428 extracts information such as "dog" or "Halloween" from the content 714 as attribute information.

[0060] In addition, the classification results of the image recognition model may be trained to include items such as "Not Applicable" to avoid including attributes that are considered inappropriate. Attributes such as "Santa Claus is standing in front of the house on a winter night" may also be acquired as explanatory text.

[0061] In S805, the prompt generation unit 427 generates a prompt for image generation to be set in the image generation AI based on the attribute information extracted from the style image. For example, the prompt generation unit 427 extracts information such as "Halloween" as a prompt for image generation to be set in the image generation AI based on the attribute information "dog" and "Halloween" extracted from the content 714.

[0062] In S806, the image generation unit 423 uses the generation model 424 in which the image generation prompt generated by the prompt generation unit 427 is set. Then, the image generation unit 423 generates an image converted into a style that harmonizes with the style of the style image, from the content image information specified in the content image input unit 415. For example, the image generation unit 423 uses the generation model 424 in which "Halloween" is set as the image generation prompt generated by the prompt generation unit 427, to generate Halloween-themed content 901 from non-Halloween-themed content 741. As a result, the style-changed image is displayed in the layout editing area on the layout data editing screen, instead of the added image.

[0063] (Layout data editing screen) 9 is a diagram showing an example of a layout data editing screen according to this embodiment. The layout data editing screen 900 is a UI screen displayed on the display 310 of the client PC 103, with the image output device A 101 as the output target (print execution target). Note that FIG. 9 shows the state after style conversion has been performed on the content 741 shown in FIG. 7.

[0064] The layout data editing screen 900 displays a template list 701 , an add image button 702 , an add text button 703 , a layout editing area 704 , a print execution button 705 , and a style conversion button 706 , just like the layout data editing screen 700 .

[0065] The layout data editing screen 900 further displays a replacement check area 910. The replacement check area 910 is displayed in association with content for which style conversion has been performed. The replacement check area 910 displays an OK button 911 and a cancel button 912. The replacement check area 910 is an area for checking whether or not to confirm replacement with the converted image after converting the style of the target image using the generation AI function. The OK button 911 is a button for accepting a user operation to confirm replacement of the content image before the style conversion with the content image after the style conversion. The cancel button 912 is a button for accepting a user operation to cancel replacement of the content image before the style conversion with the content image after the style conversion and confirm that the content image before the style conversion remains as it is. The presence of buttons for selecting whether or not to perform replacement after conversion allows the user to confirm their intention before printing, thereby preventing unnecessary printing.

[0066] (Prompt generation process for image generation) The process of generating a prompt for image generation (S805) will be described in detail. When there is one style image, the extracted attribute information may be used as is, or information indicating the atmosphere or style of the image may be prioritized from the extracted attribute information. For example, when attributes such as "winter," "dog," and "fun" are extracted, "winter" and "fun" are general attributes indicating the atmosphere or style of the image, while "dog" is a more specific attribute than the atmosphere or style of the image. For example, when an image of a person is specified as a content image, specifying "dog" as the input prompt may convert the person to a dog, which may significantly change the impression of the image. In contrast, when an image of a person is specified as a content image, specifying "winter" or "fun" as the input prompt results in less of a change in impression than "dog." Using general information as a prompt in this way allows adjustments to be made so that the impression of the original content image is not too different. To prioritize general information, a dedicated image recognition model that extracts only style and atmosphere may be used as the image recognition model for extracting attribute information in the attribute information generation unit 428.

[0067] If there are multiple style images, attribute information is acquired for each of the multiple style images. All acquired attribute information may be used to create an image generation prompt. Alternatively, the acquired attribute information may be totaled for each type, and the most common attribute information for each type may be set as the image generation prompt. If there are multiple style images, the attribute information generation unit 428 uses a dedicated image recognition model that classifies seasons, events, and image styles, and acquires one attribute information per event, image style, and season from each style image. When three images are specified as style images, it is assumed that the following attribute information is acquired for each style image:

[0068] [Table 1]

[0069] In this case, "Christmas" is the most common attribute information for all style images related to events, with two items being the most common, and so "Christmas" is used as the prompt. Similarly, for all style images, "watercolor" is the most common attribute information for painting style, with two items being the most common, and "winter" is the most common attribute information for seasons, with three items being the most common, and so "Christmas, watercolor, winter" is used as the prompt. Therefore, "Christmas, watercolor, winter" is generated as the prompt for image generation.

[0070] Furthermore, an image generation prompt may be generated that strongly reflects attributes common to many style images among the acquired attribute information. For example, consider a case where a phrase placed at the beginning of a prompt has a stronger influence on the generated image. Using the example of multiple style images described above, "winter" is acquired as a seasonal attribute from all of style images 1 to 3, and is considered to be an attribute common to all specified style images. While "watercolor" is also acquired as an attribute of painting style from two of the three style images, it is considered to be less common to all style images than "winter," and therefore it is considered more appropriate to place "winter" at the beginning of the generated prompt. Similarly, if a prompt is created in order of common attribute information, when there are multiple style images, "winter, Christmas, watercolor, anime style" may be considered as an example of a generated prompt.

[0071] Furthermore, when attribute information is acquired as text, the attribute information may be further summarized and used as a prompt for generating an image. For example, a generation model that receives text as input, such as GPT-3 disclosed in Non-Patent Document 3, and outputs text may be given an input such as "Please output a phrase that expresses the common atmosphere from the following three sentences," and the output may be used as a prompt for generating an image.

[0072] Furthermore, phrases that are likely to be used in prompts may be set in advance as prompt templates and added to the prompt generated from the attribute information. For example, phrases such as "masterpiece" and "best quality," which are phrases that indicate high-quality images, may be stored and added to the beginning of the prompt generated from the attribute information. A template corresponding to an expected conversion method may also be set, and the content of the template specified by a user operation may be added to the prompt. For example, if conversion to an anime or illustration style is frequently used, the following settings may be made and added. That is, phrases such as "a sketch of" or "a illustration of" may be set as prompt templates, and the user may specify "anime style" or the like as an option during conversion, which may then be added to the prompt.

[0073] Furthermore, when generating a prompt from attribute information, multiple prompts may be generated and multiple images may be generated for the user to select from. For example, even if the same content image is specified, different images may be generated depending on the order of the words in the prompt. Therefore, multiple images may be generated by changing the order of the words in the prompt. Also, multiple prompts may be generated by adding prepared prompt templates such as "anime style," "live action style," and "abstract painting" to the prompt generated from the attribute information. Then, multiple images may be generated from each prompt and content image.

[0074] As described above, in this embodiment, when generating an image suitable for an image in layout data, an image generation prompt is automatically created and set from a style image representing the style included in the template. Therefore, it is possible to easily set an appropriate prompt for obtaining desired content that harmonizes with the content already placed in the template. This eliminates the need to set information describing the content for each of the multiple images included in various templates before creating a poster, flyer, or the like. This allows for the generation of an image with a style that harmonizes with the style of the image included in the template.

[0075] In addition, when there are multiple editing targets for the generation AI, it becomes easier for users to issue prompts for multiple editing targets.Furthermore, it also reduces the burden on content providers of setting prompts for each image content.

[0076] <<Embodiment 2>> In this embodiment, an aspect will be described in which attribute information and prompts for individual style images can be modified by user operations. Note that in this embodiment, the differences from the first embodiment will be mainly described.

[0077] (Functional configuration of information processing system) 10 is a block diagram showing an example of the functional configuration of an information processing system according to this embodiment. In the information processing system of this embodiment, an image output device A101 is the output target (print execution target). The client PC 103 of this embodiment has the same functional units 411 to 416 as the client PC 103 of the first embodiment, and further has an attribute information operation request unit 1001. The server 105 of this embodiment has the same functional units 421 to 429 as the server 105 of the first embodiment, and further has an attribute information operation unit 1002.

[0078] The attribute information operation request unit 1001 requests the attribute information operation unit 1002 to operate the attribute information. The attribute information operation unit 1002 operates the attribute information.

[0079] (Layout data) 11 is a diagram showing an example of layout data stored in the layout data DBs 411 and 422 according to this embodiment. The layout data DB 1100 according to this embodiment has attribute information 1101 in addition to information about each of the items 601 to 605 that the layout data DB 600 according to the first embodiment has. In the attribute information 1101, attribute information for each individual image is added. A prompt, which is overall attribute information, is also added to ID-G. In addition to the attribute information acquired from the dog image ID-D, which is a style image, a prompt is set to which "masterpiece" and "best quality" prepared in the template are added.

[0080] (Layout data editing screen) 12 is a diagram showing an example of a layout data editing screen according to this embodiment. Similar to the layout data editing screen 700, the layout data editing screen 1200 is a UI screen displayed on the display 310 of the client PC 103, and is targeted for output (print execution) to the image output device 101. In FIG. 12, it is assumed that the template 710 in the template list 701 is selected. It is also assumed that the content 714 is selected to modify the attribute information of the content 714, and the style image attribute display section 1201 is displayed, and the prompt information input section 1205 is displayed to update the prompt.

[0081] Similar to the layout data editing screen 700, the layout data editing screen 1200 displays a template list 701, an add image button 702, an add text button 703, a layout editing area 704, and a print button 705. The layout data editing screen 1200 also displays a style conversion button 706 and a generation AI function area 707. The generation AI function area 707 displays a style image target checkbox 708 and a content image target checkbox 709. The generation AI function area 707 further displays a style image attribute display section 1201, an acquire attribute button 1202, and a modify attribute button 1203. The layout data editing screen 1200 displays an accordion button 1206 for style conversion. When a user operates the accordion button 1206, a details box 1204 of prompt information in the layout data is displayed. The details box 1204 displays a prompt information input section 1205, an acquire prompt information button 1207, and an update prompt information button 1208.

[0082] The style image attribute display unit 1201 displays attribute information for each style image. When the style image target check box 708 in the generation AI function area 707 set for each image is selected and turned ON by a user operation and the attribute acquisition button 1202 is pressed, the following processing is performed. That is, the attribute information operation request unit 1001 requests the attribute information generation unit 428 of the server 105 to generate attribute information for the style image, and stores the generated attribute information of the style image in the layout data DB 422. The layout data editing unit 412 reads the information stored in the layout data DB 422 and displays the attribute information in the style image attribute display unit 1201. The user can modify the attribute information displayed in the style image attribute display unit 1201 by keyboard input, etc. After the attribute information is modified, when the attribute modification button 1203 is pressed by a user operation, the attribute information of the style image stored in the layout data DB 422 is updated.

[0083] Furthermore, a prompt information details box 1204 in the layout data is displayed when the user selects an accordion button 1206 with a mouse. When the user presses a prompt information acquisition button 1207, the prompt information input unit 1205 requests the attribute information operation request unit 1001 to acquire a prompt from the attribute information operation unit 1002 based on the style image specified in the style image target check box 708. The attribute information operation unit 1002 acquires attribute information of the target style image from the layout data DB. Furthermore, the attribute information operation unit 1002 acquires attribute information of the target style image from the attribute information generation unit 428 for style images for which no attribute information exists. The prompt generation unit 427 then generates a prompt based on the attribute information and updates the prompt information in the layout data DBs 411 and 422 with the generated prompt. The layout data editing unit 412 reads the information in the layout data DB, and the prompt information generated based on the attribute information of the style image is displayed in the prompt information input unit 1205. Further, if the prompt information needs to be corrected, the user can do so by inputting a prompt into the prompt information input section 1205. Then, when the user presses the prompt information update button 1208, the attribute information operation request section 1001 requests the attribute information operation section 1002 to update the prompt information in the layout data DB 422 with the information input into the prompt information input section 1205. Thereafter, when the user presses the style conversion button 706, the target content image is converted with the prompt information updated by the user's input. This allows the user to fine-tune the content of the conversion by their own work.

[0084] (Image generation processing) FIG. 13 is a flowchart showing the flow of image generation processing according to this embodiment. FIG. 13 shows the flow of processing in which an image matching an image already arranged on the layout data editing screen 1200 is generated by a generation AI. That is, all content included in a template is set in advance as a style image, and the flow of processing in which the style of an added image is converted so as to harmonize with the style of the style image is shown. The flowchart shown in FIG. 13 is realized, for example, by the CPU 301 reading a program stored in the ROM 302 into the RAM 303 and executing it.

[0085] 12 at the timing when, for example, the style conversion button 706, the attribute modification button 1203, or the prompt information update button 1208 is pressed on the layout data editing screen 1200. In this embodiment, unlike the first embodiment, a prompt may be stored in advance in the layout data DB 410 by pressing the prompt information acquisition button 1207 or the prompt information update button 1208.

[0086] In S1301, the server 105 determines whether a prompt is stored in the layout data DB. If the determination result indicates that the prompt is stored in the layout data DB (YES in S1301), the process proceeds to S806. On the other hand, if the determination result indicates that the prompt is not stored in the layout data DB (NO in S1301), the process proceeds to S1302. For example, if a user operation is performed on the style information acquisition button 1107 or the prompt information update button 1208, the prompt information is saved in the layout data DBs 422 and 411, and the process proceeds to S806. On the other hand, if a user operation is not performed on either the prompt information acquisition button 1207 or the prompt information update button 1208, the prompt information is not saved in the layout data DBs 422 and 411. Therefore, the process proceeds to S1302.

[0087] In S1302, the server 105 determines whether or not the image specified in the content image input unit 404 has attribute information. If the determination result indicates that the image specified in the content image input unit 404 has attribute information (YES in S1302), the process proceeds to S805. On the other hand, if the determination result indicates that the image specified in the content image input unit 404 does not have attribute information (NO in S1302), the process proceeds to S804. For example, if a user operation is performed on the attribute acquisition button 1202 or the attribute modification button 1203, attribute information is stored in the layout data DBs 422 and 411, and the process proceeds to S805. On the other hand, if a user operation is not performed on either the attribute acquisition button 1202 or the attribute modification button 1203, attribute information is not stored in the layout data DBs 422 and 411, and the process proceeds to S802.

[0088] In addition, in S804 to S806, the same processes as those in the first embodiment are executed, and therefore detailed explanations are omitted.

[0089] As described above, according to this embodiment, the user can obtain attribute information and prompts for individual images and make fine adjustments or other corrections before generating the image, making it easier to generate a more desired image.

[0090] <Other embodiments> The present disclosure can also be realized by executing the following process. That is, a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0091] Although the above description has been given of an application in which the present invention is applied to a layout data creation application as an example of an application, the present invention is not limited to this, and can also be applied to any application that has an image layout function similar to the layout data creation application.

[0092] In the above, a PC, which is an information processing device, has been described as an example of the client PC 103, but the present invention is not limited to this. For example, any information processing device (terminal) that can be used in a similar manner, such as a mobile phone, a mobile information terminal, a digital still camera, a digital video camera, a portable music player, a game, a set-top box, or an Internet appliance, can also be applied.

[0093] Although the above description has been given taking Ethernet as an example of the network configuration, the present invention is not limited to this and may be any other network configuration such as wireless LAN, IEEE1394, Bluetooth, etc.

[0094] Although the above describes an example in which all content included in a template is set as a style image in advance, this is not limiting. For example, some of all content included in a template may be set as a style image in advance, or not all content included in a template may be set as either a style image or a content image. In this case, all content included in the template may be individually set as a style image or a content image.

[0095] The above describes an aspect in which a dedicated image recognition model is used for each of the season, event, style, or atmosphere to extract information indicating the season, event, style, or atmosphere as attribute information, but this is not limiting. For example, when a person or the like is present, a dedicated image recognition model may be used to recognize facial expressions such as joy and sadness, thereby extracting information indicating facial expressions as attribute information. Furthermore, when a person or the like is present, a dedicated image recognition model may be used to recognize a happy state, a sad state, etc., thereby extracting information indicating emotions as attribute information.

[0096] In the above description, when attribute information or a prompt is modified, the modified attribute information or prompt is stored in the layout data DBs 422 and 411. However, the present invention is not limited to this. For example, attribute information extracted from a style image may be associated with the corresponding style image and stored in the DBs 422 and 411, and the attribute information stored in the DBs 422 and 411 may be used when converting the style of content based on the style image. For example, a prompt created from attribute information may be linked to a style image corresponding to the attribute information and stored in DBs 422 and 411, and the prompt stored in DBs 422 and 411 may be used when converting the style of content based on the style image.

[0097] The disclosure of this embodiment includes the following configuration examples. (Configuration 1) An information processing device for causing a generation AI to generate content, An acquisition means for acquiring a base image that is the basis of the content that the generation AI is to generate and a style image that represents the style of the content; an extracting means for extracting attribute information indicating the style represented by the style image from the style image acquired by the acquiring means; a setting means for setting a prompt for the generation AI to generate the content based on the base image, to the generation AI based on the attribute information extracted by the extraction means; An information processing device comprising: (Configuration 2) The extraction means extracts the attribute information by performing image recognition using an image recognition model. 2. The information processing device according to configuration 1, (Configuration 3) When the acquisition unit acquires a plurality of style images, the extraction unit extracts the attribute information for each of the plurality of style images; The setting means sets the prompt based on the plurality of pieces of attribute information extracted by the extraction means. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The setting means classifying the plurality of pieces of attribute information extracted by the extraction means by type; Set the prompt based on the most common attribute information for each classified type. 4. The information processing device according to configuration 3. (Configuration 5) The extraction means extracts the attribute information including preset setting information, The setting means sets the prompt based on attribute information including the setting information extracted by the extraction means. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) further comprising a receiving means for receiving a user operation, The setting means generating a plurality of prompts from the plurality of attribute information; A prompt selected by the user operation accepted by the accepting means is set from among the plurality of generated prompts. 5. The information processing device according to configuration 3 or 4. (Configuration 7) further comprising a storage means for storing the prompt; The setting means sets the prompt corresponding to the style image stored by the storage means when the prompt corresponding to the style image is stored in the storage means. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The method further includes a modifying unit that modifies the prompt stored by the storing unit based on a user operation, The setting means sets the prompt modified by the modifying means. 8. The information processing device according to configuration 7. (Configuration 9) The modifying means modifies the prompt by the user's operation via a UI screen displayed on a display unit. 9. The information processing device according to configuration 8. (Configuration 10) Further, a storage means for storing the attribute information is provided. When the attribute information corresponding to the style image is stored in the storage means, the setting means sets the prompt based on the attribute information corresponding to the style image stored by the storage means. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) The attribute information storing unit further includes a modifying unit that modifies the attribute information stored by the storing unit based on a user operation. The setting means sets the prompt based on the attribute information corrected by the correcting means. 11. The information processing device according to configuration 10. (Configuration 12) The modifying means modifies the attribute information by the user's operation via a UI screen displayed on a display unit. 12. The information processing device according to configuration 11. (Configuration 13) The system further includes a receiving unit for receiving a request for conversion by the generation AI for which the prompt is set by the setting unit for the input new content. 13. The information processing device according to any one of configurations 1 to 12. (Configuration 14) 14. The information processing apparatus according to configuration 13, wherein the accepting means accepts a request for the conversion each time the new content is input. (Configuration 15) The receiving means performs the conversion when receiving a user instruction. 14. The information processing device according to configuration 13. (Configuration 16) The extraction means extracts at least one of information indicating a season, information indicating an event, information indicating a style of painting, information indicating an atmosphere, information indicating an emotion, and information indicating an expression as the attribute information. 16. The information processing device according to any one of configurations 1 to 15. (Configuration 17) Further, a generating means for generating a deliverable incorporating the content generated by the generating AI is provided. The acquiring means acquires the style image from a template prepared for the product. 17. The information processing device according to any one of configurations 1 to 16. (Configuration 18) An information processing method for causing a generation AI to generate content, An acquisition step of acquiring a base image that is the basis of the content to be generated by the generation AI and a style image that represents the style of the content; an extraction step of extracting attribute information indicating the style represented by the style image from the style image acquired in the acquisition step; a setting step of setting a prompt for the generation AI to generate the content based on the base image, in the generation AI based on the attribute information extracted in the extraction step; An information processing method comprising: (Configuration 19) 19. A program for causing a computer to execute each step of the information processing method according to claim 18.

Claims

1. An information processing device for causing a generation AI to generate content, An acquisition means for acquiring a base image that is the basis of the content to be generated by the generation AI and a style image that represents the style of the content; an extracting means for extracting attribute information indicating the style represented by the style image from the style image acquired by the acquiring means; a setting means for setting a prompt for the generation AI to generate the content based on the base image, based on the attribute information extracted by the extraction means; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the extracting means extracts the attribute information by performing image recognition using an image recognition model.

3. When the acquisition unit acquires a plurality of style images, the extraction unit extracts the attribute information for each of the plurality of style images; The setting means sets the prompt based on the plurality of pieces of attribute information extracted by the extraction means.

2. The information processing apparatus according to claim 1, wherein:

4. The setting means classifying the plurality of pieces of attribute information extracted by the extraction means by type; Set the prompt based on the most common attribute information for each classified type.

4. The information processing apparatus according to claim 3,

5. The extraction means extracts the attribute information including preset setting information, The setting means sets the prompt based on attribute information including the setting information extracted by the extraction means.

2. The information processing apparatus according to claim 1, wherein:

6. further comprising a receiving means for receiving a user operation, The setting means generating a plurality of prompts from the plurality of attribute information; A prompt selected by the user operation accepted by the accepting means is set from among the plurality of generated prompts.

4. The information processing apparatus according to claim 3,

7. further comprising a storage means for storing the prompt; The setting means sets the prompt corresponding to the style image stored by the storage means when the prompt corresponding to the style image is stored in the storage means.

2. The information processing apparatus according to claim 1, wherein:

8. The method further includes a modifying unit that modifies the prompt stored by the storing unit based on a user operation, The setting means sets the prompt modified by the modifying means.

8. The information processing apparatus according to claim 7,

9. The modifying means modifies the prompt by the user's operation via a UI screen displayed on a display unit.

9. The information processing apparatus according to claim 8,

10. Further, a storage means for storing the attribute information is provided. When the attribute information corresponding to the style image is stored in the storage means, the setting means sets the prompt based on the attribute information corresponding to the style image stored by the storage means.

2. The information processing apparatus according to claim 1, wherein:

11. The attribute information storing unit further includes a modifying unit that modifies the attribute information stored by the storing unit based on a user operation. The setting means sets the prompt based on the attribute information corrected by the correcting means.

11. The information processing apparatus according to claim 10,

12. The modifying means modifies the attribute information by the user's operation via a UI screen displayed on a display unit.

12. The information processing apparatus according to claim 11,

13. The system further includes a receiving unit for receiving a request for conversion by the generating AI for which the prompt is set by the setting unit for the input new content.

2. The information processing apparatus according to claim 1, wherein:

14. 14. The information processing apparatus according to claim 13, wherein said accepting unit accepts the decision as to whether or not the conversion is necessary each time the new content is input.

15. The receiving means performs the conversion when receiving a user instruction.

14. The information processing apparatus according to claim 13,

16. The extraction means extracts at least one of information indicating a season, information indicating an event, information indicating a style of painting, information indicating an atmosphere, information indicating an emotion, and information indicating an expression as the attribute information.

2. The information processing apparatus according to claim 1, wherein:

17. Further, a generating means for generating a deliverable incorporating the content generated by the generating AI is provided, The acquiring means acquires the style image from a template prepared for the product.

2. The information processing apparatus according to claim 1, wherein:

18. An information processing method for causing a generation AI to generate content, An acquisition step of acquiring a base image that is the basis of the content to be generated by the generation AI and a style image that represents the style of the content; an extraction step of extracting attribute information indicating the style represented by the style image from the style image acquired in the acquisition step; a setting step of setting a prompt for the generation AI to generate the content based on the base image based on the attribute information extracted in the extraction step; An information processing method comprising:

19. A program for causing a computer to execute each step of the information processing method according to claim 18.

Images