Generation support device, generation support program, and generation support method

Abstract

To provide a generation support device, a generation support program, and a generation support method that enable users to easily generate target images.SOLUTION: A server device 1 of an evaluation system which is communicably connected to a user terminal via a communication network is a generation support device that supports the generation of a resultant image. The server device includes: a generation information acquisition unit that acquires, from a user, generation information including at least style information related to a style of the resultant image and an element image constituting a part of the resultant image; and a resultant image generation unit that inputs text information generated based on the generation information into a generation model and generates the resultant image based on output information output from the generation model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a generation support device, a generation support method, and a generation support program. [Background technology]

[0002] Recently, images have been generated by various methods. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7169027 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, Patent Document 1 proposes a technique for generating character images using machine learning.

[0005] However, the technology of Patent Document 1 is only capable of generating images of characters in arbitrary poses, and cannot be applied to generating a variety of images.

[0006] The present invention has been made in view of the above background, and has an object to enable a user to easily generate a desired image. [Means for solving the problem]

[0007] In order to solve the above problems, the generation assistance device disclosed herein is characterized by comprising a generation information acquisition unit that acquires generation information from a user, the generation information including at least style information regarding the style of a result image and an element image that constitutes part of the result image, and a result image generation unit that inputs text information generated based on the generation information into a generation model and generates the result image based on output information output from the generation model.

[0008] Other problems and solutions disclosed in this application will be made clear in the description of the preferred embodiments of the invention and the drawings. Effect of the Invention

[0009] According to the present invention, a user can easily generate a desired image. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an example of the overall configuration of an evaluation system according to an embodiment of the present invention; [Diagram 2] 2 is a diagram illustrating an example of a hardware configuration of a server device 1 according to the embodiment. FIG. [Diagram 3] 2 is a diagram illustrating an example of a functional configuration of a server device 1 according to the embodiment. FIG. [Figure 4] 13 is a diagram showing an example of basic information stored in a generation information storage unit 131. FIG. [Diagram 5] 13 is an example of a screen on which the generation information acquisition unit 111 acquires position information of a partial image and a material image. [Figure 6] FIG. 11 is a diagram illustrating an example of a process of the server device 1 according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Summary of the Invention> [Item 1] A generation support device that supports generation of a result image, a generation information acquisition unit that acquires generation information including at least style information related to a style of the resultant image and an element image that constitutes a part of the resultant image from a user; a resultant image generation unit that inputs text information generated based on the generation information into a generation model and generates the resultant image based on output information output from the generation model; A generation assistance device comprising: [Item 2] the generation information acquisition unit further acquires position information of the element images in the resultant image; 2. The generation assistance device according to item 1, [Item 3] The style information includes information of a web address, the generation information acquisition unit determines the style information based on web information included in the website specified by the web address as the style information; 3. The generation assistance device according to item 1 or 2, [Item 4] the information acquisition unit accepts upload or selection of the element image, and acquires the position information based on an arrangement of the element image in a frame of the resultant image; 3. The generation assistance device according to item 2, characterized in that [Item 5] the information acquisition unit acquires the generation information through input by the user in a chat format; 3. The generation assistance device according to item 1 or 2, [Item 6] when acquiring the input in the chat format, the information acquisition unit presents to the user a suggestion of information required as the generation information; 6. The generation assistance device according to item 5, [Item 7] A generation assistance program for assisting in the generation of a resultant image, The processor: a generation information acquisition step of acquiring generation information including at least style information related to a style of the resultant image and an element image constituting a part of the resultant image from a user; an image generation step of inputting text information generated based on the generation information into a generation model, and generating the resultant image based on output information output from the generation model; A generation support program that executes the above. [Item 8] A method for assisting in the generation of a result image, comprising: The processor: a generation information acquisition step of acquiring generation information including at least style information related to a style of the resultant image and an element image constituting a part of the resultant image from a user; an image generation step of inputting text information generated based on the generation information into a generation model, and generating the resultant image based on output information output from the generation model; A generation assistance method for executing the above.

[0012] 1 is a diagram showing an example of the overall configuration of an evaluation system according to an embodiment of the present invention. The generation support system of this embodiment is configured to include a server device 1. The server device 1 is communicably connected to a user terminal 3 via a communication network 2. The communication network 2 is, for example, the Internet, and is constructed by a public telephone line network, a mobile phone line network, a wireless communication path, Ethernet (registered trademark), or the like.

[0013] ==Server device 1== The server device 1 may be, for example, a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing. In this embodiment, one server device is illustrated for convenience of explanation, but the present invention is not limited to this and may be multiple servers.

[0014] ==User terminal 3== The user terminal 3 is a computer operated by a user who generates an image. For example, the user terminal 3 is a smartphone, a tablet computer, a personal computer, etc. The user can access the server device 1 by, for example, an application or a web browser executed on the user terminal 3.

[0015] FIG. 2 is a diagram showing an example of the hardware configuration of the server device 1. Note that the illustrated configuration is an example, and other configurations may be used. The server device 1 includes a processor 101, a memory 102, a storage device 103, a communication interface 104, an input device 105, and an output device 106. The storage device 103 is, for example, a hard disk drive, a solid state drive, or a flash memory that stores various data and programs. The communication interface 104 is an interface for connecting to the communication network 2, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 105 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, etc. that input data. The output device 106 is, for example, a display, a printer, a speaker, etc. that output data. Each functional unit of the server device 1, which will be described later, is realized by the processor 101 reading a program stored in the storage device 103 into the memory 102 and executing it, and each storage unit of the server device 1 is realized as part of the storage area provided by the memory 102 and the storage device 103.

[0016] Fig. 3 shows a functional configuration of the server device 1. As shown in Fig. 3, the server device 1 includes a generation information storage unit 131, a resultant image information storage unit 132, and other storage units, as well as a generation information acquisition unit 111 and a resultant image generation unit 112.

[0017] The generation information storage unit 131 and the resultant image information storage unit 132 will be described below.

[0018] The generation information storage unit 131 stores information (hereinafter, referred to as generation information) used to generate a result image (an image generated by the server device 1), as shown in FIG. 4 as an example. The generation information may include, for example, information related to style (including text information, web address, web information, etc.). The generation information may also include an element image that is the basis for the configuration of a part of the result image. The element image is, for example, a partial image that is an image of the subject of the result image (for example, a person or an object, which is described as the main content of the image when the generation information acquisition unit 111 described below generates text to be input to the generation model), and if the subject of the partial image is an image of an object, it may include, for example, an image of a product, an image including a product, an image of the appearance of a product container, an outer box, etc. The element image may also include a material image that is not the subject of the result image. The generation information may include, for example, information on the position of the partial image and the material image in the result image, but is not limited thereto.

[0019] The style refers to the style in the design of the resultant image generated by the generation support device, and represents, for example, aesthetic attributes and characteristics such as the requirements of the resultant image (such as objects, people, landscapes, and other elements contained in the image), concept (target, narrative, etc.), color, texture (such as the texture of the image surface that is perceived by the visual senses), layout (such as the arrangement and relative positions of elements), font, and shape (sharp angles, rounded shapes, straight lines, curves, etc.).

[0020] The material images are images of hands, faces, plants, everyday items, stands, geometric shapes such as circles, triangles, and squares, or free shapes not bound by geometric rules, and are images that serve as the basis for a part of the resulting image. The material images may also include images (templates) that serve as the basis for the background of the image to be generated, but are not limited to these.

[0021] The resultant image information storage unit 132 stores the resultant image generated by the resultant image generating unit 112 .

[0022] The following describes each processing unit of the generation information acquisition unit 111 and the resultant image generation unit 112.

[0023] As an example, the generation information acquisition unit 111 acquires generation information including style information related to the style of the result image, element images constituting the result image, and position information of the element images in the result image, which are necessary for generating a result image, from the user terminal 3 via the communication network 2. The generation information acquisition unit 111 stores the acquired generation information in the generation information storage unit 131. The communication in the transmission and reception may be either wired or wireless, and any communication protocol may be used as long as the communication between them can be performed.

[0024] The generation information acquisition unit 111 may acquire the generation information as text information. The generation information acquisition unit 111 may acquire a sentence indicating a resultant image to be generated by a user's input operation, or may acquire one or more words. The generation information acquisition unit 111 may also present a sentence or word indicating the style of the resultant image to be generated to the user, and acquire the sentence or word selected by the user as the generation information.

[0025] The generation information acquisition unit 111 may acquire information of a web address (URL, etc.) as the generation information. The generation information acquisition unit 111 may acquire web information included in a website specified by the web address, determine a style, and set the generated information as the generation information. The web information may be text information, image information, video information, code information (code constituting a website, which may be in a format such as HTML, CSS, Javascript, etc., but is not limited to these) included in the website. The generation information acquisition unit 111 may determine a style of a target, a concept, etc., based on the text information, for example. In this case, the generation information acquisition unit 111 may perform a morphological analysis of the text information, and determine a style of a target, a concept, etc., based on information on the words included and the number of words, but is not limited to these methods. The generation information acquisition unit 111 may determine a style of a color, texture, font, shape, etc., based on the image information, video information, or code information, etc. In this case, the generation information acquisition unit 111 may analyze the image information or video information and determine the style from the predominant colors, textures, fonts, shapes, etc., or may determine the style based on information contained in the code information, such as the colors, textures, shapes, and fonts used on the web of the web background image, but is not limited to these methods.

[0026] The generation information acquisition unit 111 acquires element images that are the basis for forming a part of the resultant image. The generation information acquisition unit 111 may accept uploading of element images. Alternatively, as shown in an example in Fig. 5, the generation information acquisition unit 111 may store a material image (e.g., 201 in Fig. 5) in the server device 1, present the material image to the user terminal 3, accept a selection operation of the material image by the user, and acquire the material image selected by the user as generation information.

[0027] As an example, the generation information acquisition unit 111 acquires position information of the element image in the result image. The position information indicates the coordinates of the element image in the result image, and may be, for example, XY coordinates with a specific position such as a specific corner of the result image as the origin, and may be XY coordinates of the center of the element image, for example. In this case, the generation information acquisition unit 111 presents a frame (for example, 202. The frame may be horizontally long, square, vertically long, or the like, but is not limited to these) according to the shape of the result image to the user terminal 3, as shown in FIG. 5 as an example. The generation information acquisition unit 111 acquires information on the user's operation on the user terminal 3 in the frame, and acquires position information of the material image in the result image. In this case, the generation information acquisition unit 111 may accept, for example, a drag-and-drop operation by the user, acquire arrangement information of the partial image (203) or the material image (204), and acquire position information of the element image. The generation information acquisition unit 111 may also accept enlargement, reduction, rotation, inversion, deformation, and the like of the element image. Furthermore, the generation information acquisition unit 111 may acquire the anteroposterior relationship of a plurality of element images (for example, layer information) as the position information. Alternatively, the position information may be information on the positional relationship between element images (such as that a material image is located below a partial image).

[0028] The generation information acquisition unit 111 may acquire the generation information in a chat format. In this case, the generation information acquisition unit 111 may divide the text information acquired in the chat format into two words by morphological analysis or the like, and acquire the information of the words as the generation information. In this case, the generation information acquisition unit 111 may provide assistance to the user, such as "Please upload a subject image" or "Please tell me a reference website," with respect to the generation information acquired from the user, so that the user can easily recognize the necessary generation information. In this case, the generation information acquisition unit 111 may provide the user with guidance on information not acquired from the user or information that could not be determined from the acquired generation information from a list of generation information required for image generation prepared in advance, but is not limited to this method.

[0029] The generation information acquisition unit 111 generates a prompt or prerequisite conditions (collectively referred to as prompt information in this specification) to be input to an image generation model based on the acquired generation information. The prerequisite conditions may include, but are not limited to, information such as image size, frame shape, file size, and resolution. The prompt generated by the generation information acquisition unit 111 includes at least text representing a style. The generation information acquisition unit 111 generates the prompt information using, for example, a feature extraction module and a language model. Note that the generation information acquisition unit 111 may generate one or more pieces of prompt information, present it to the user, and accept selection or editing of a prompt.

[0030] When generating a prompt, the generation information acquisition unit 111 may change the structure of the generated prompt depending on the type of generative model used by the result image generation unit 112 for image generation. The generation information acquisition unit 111 may generate, for example, a sentence-type prompt or a prompt in the form of a list of words. In addition, the generation information acquisition unit 111 may generate a prompt in the form of making the generative model recognize important words by a method of indicating the importance of words, such as enclosing important words in parentheses, making the order of words appear at the beginning of the prompt, or including multiple important words.

[0031] The prompt generated by the generation information acquisition unit 111 includes at least text representing a style. The generation information acquisition unit 111 may generate a plurality of prompts, or may impart a certain degree of randomness to the text included in the prompt. For example, the generation information acquisition unit 111 imparts a certain degree of randomness to the text included in the prompt, depending on the semantic distance or similarity to the text representing the style. Specifically, when the generation information acquired by the user's input or the like includes a plurality of pieces of style information related to, for example, the "sea", the generation information acquisition unit 111 generates a prompt by including in the prompt other words that are close in meaning to the "sea" or have a high similarity. The result image generation unit 112, which will be described later, generates a result image that is closer to the image that the user wishes to generate by using these prompts. Conversely, when the generation information acquired by the user's input or the like includes little style information related to the "sea", the generation information acquisition unit 111 generates a prompt by including in the prompt other words that are far from the semantic meaning to the "sea" or have a low similarity to the "sea". The result image generating unit 112, which will be described later, generates a result image using these prompts, making it easier for the user to consider the direction of the style of the result image when the user does not yet have an image of the ocean, etc. Note that giving a certain degree of randomness to the text included in the prompts may have other effects.

[0032] The generation information acquisition unit 111 may acquire additional generation information for the first result image generated by the result image generation unit 112. The generation information additionally acquired by the generation information acquisition unit 111 is used for modifying or adding to the prompt used when the first result image was generated, and is used when the result image generation unit 112 generates the second result image.

[0033] As an example, the result image generating unit 112 generates a result image based on at least one of style information, element image, and position information. For example, the result image generating unit 112 inputs prompt information generated by the generation information acquiring unit 111 based on at least one of style information, element image, and position information to a generation model, and acquires an image output by the generation model. The result image generating unit 112 may use the image output by the generation model as a result image, or may generate a result image by performing processing or the like based on the output image. The result image generating unit 112 presents the generated result image to the user. The user can download the presented image.

[0034] The generative model used by the resultant image generation unit 112 to generate the resultant image may be implemented in the server device 1 or may be implemented in another server accessible via the communication network 2, but is not limited to this. For this reason, when the generative model is implemented in the server device 1, the resultant image generation unit 112 inputs prompt information to the generative model, and when the generative model is implemented in another server, the resultant image generation unit 112 transmits the prompt information to the generative model via the communication network 2. In this specification, the expression "inputting prompt information to the generative model" is used to include the case where the prompt information is transmitted to the generative model.

[0035] The generative model may be, for example, a model that receives a specific input vector or random noise given as an input and generates an image from the information. The generative model includes, for example, a generator. The generator converts the input information into appropriate features or patterns and converts them into an image. The generator is constructed using, for example, a Convolutional Neural Network (CNN), a Transformer, or other deep learning architecture, but other architectures can also be used. The generative model also includes, for example, a discriminator. The discriminator discriminates whether an image is a real image or a fake image generated by the generator. The discriminator is constructed using, for example, a network such as a CNN, but is not limited to this. The generative model includes, for example, an adversarial network (GAN). The adversarial network learns to make the generator generate more realistic images and at the same time to improve the discriminator's ability to distinguish between real images and fake images.

[0036] The resultant image generating unit 112 may generate two or more resultant images. Furthermore, the resultant image generating unit 112 presents the generated resultant images to the user.

[0037] When the result image generating unit 112 presents the generated multiple result images to the user terminal 3, the result image generating unit 112 may receive a selection operation from the user at the user terminal 3 of an image that is close to the result image to be generated from the multiple images or an image that is different from the result image to be generated, and may generate a further result image based on the result image selected by the selection operation. In this case, the result image generating unit 112 may generate a result image B similar to the result image A based on the characteristics of the result image A selected as being close to the result image to be generated from the image. As a specific process, the generation information acquiring unit 111 may modify the information of the prompt input to the generative model when generating the result image A selected by the selection operation, or generate a prompt indicating regeneration of an image or variation similar to the selected result image A, and input these prompts again to the generative model to generate the result image, but is not limited to this method.

[0038] The result image generating unit 112 may generate a second result image for the generated result image (first result image) when the generation information acquiring unit 111 acquires additional information from the user. In this case, the result image generating unit 112 inputs the prompt information input to the generative model when generating the first result image and the prompt information generated by the generation information acquiring unit 111 based on the additional information to the generative model, and generates the second result image based on the output information output by the generative model.

[0039] FIG. 6 is a diagram illustrating an example of the process of the generation support device according to the present embodiment.

[0040] The server device 1 acquires generation information from a user (1001). The server device 1 generates a prompt based on the acquired generation information (1002). The server device 1 inputs the prompt to a generation model (1003). The server device 1 acquires output information (result image) of the generation model (1004). The server device 1 presents the output information to the user (1005).

[0041] Other examples are described below.

[0042] The server device 1 may, for example, perform pre-processing of the partial image acquired by the generation information acquisition unit 111. The server device 1 may, for example, determine the main subject of the partial image and remove the background other than the main subject portion. In addition, the generation information acquisition unit 111 may, for example, emphasize the main subject from the partial image.

[0043] As a pre-processing step, the server device 1 may, for example, determine the relationship between the subject and the camera position for a subject included in a partial image, detect the camera angle, and generate a prompt accordingly for the generation information acquisition unit 111. The prompt may include, but is not limited to, a prompt specifying the angle at which the subject is displayed in the resultant image.

[0044] Note that the generation information acquisition unit 111 may acquire position information in the image to be generated for the partial image after the above-mentioned pre-processing has been performed, or may generate a prompt.

[0045] The server device 1 may suggest a style to the user based on marketing information. The marketing information may include information on the product that is the subject of the result image, industry information, results of marketing research, etc., previously acquired information, and information on past sales of the subject product and sales of similar products, etc., acquired from the user. For example, the server device 1 suggests to the user a style determined from a sales website or advertisement image of a similar product with high sales volume for the subject product of the image to be generated based on information on sales volume of similar products. In this case, the server device 1 may present to the user information on the web address of the sales website of the similar product with high sales volume or text information (e.g., "luxury" or "natural") to be included in the prompt generated by the generation information acquisition unit 111 as a style, or may include the information in the prompt used for image generation.

[0046] The server device 1 may recommend a style to the user based on the result image generated by the user in the past using the server device 1 or the information of the prompt used when generating the result image. For example, the server device 1 may analyze the result image generated by the user in the past, or may determine the style based on the text information included in the prompt, present the most frequently detected style to the user terminal 3, and obtain a selection operation of whether or not to use the style in generating the result image. Specifically, for example, when the server device 1 determines that the user has generated only realistic images in the past, it may present a question such as "Do you want to generate a realistic image? Yes No" to the user terminal 3 by chat or the like, obtain the selection operation of the user, and generate a prompt based on the answer selected by the user.

[0047] The server device 1 may generate information related to product sales, not limited to images. Information generated by the server device 1 includes, but is not limited to, banner advertisement images for advertising products and campaigns, effective catchphrases and catchy copy that concisely express the features of products and brands, product descriptions that are text information that describe detailed descriptions and features of products, designs and layouts used on the top pages of EC sites that sell products, category page designs that are designs and display methods of product category pages, landing page designs for emphasizing specific campaigns and products, images and catchphrases for social media and advertising platforms, and the like. When the server device 1 generates the above-mentioned information, the generation information acquisition unit 111 generates a prompt based on the information acquired, and if it is an image or design, the result image generation unit 112 inputs the prompt into an image generation model, and if it is text information, inputs the prompt into a text generation model (for example, a large-scale language model such as ChatGPT) to generate the information.

[0048] When acquiring an element image constituting a part of the resultant image, the server device 1 may acquire an image included in a website specified by a specified web address as the element image. The server device 1 may acquire all images included in the website as element images and store them in the generation information storage unit 121, or may accept a user's selection operation for an image to be acquired as the generation information from among the images included in the website, and store the selected image as the element image.

[0049] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0050] The device described in this specification may be realized as a single device, or may be realized by a plurality of devices (e.g., cloud servers) some or all of which are connected via a communication network 2. For example, the processor 101 and the storage device 103 of the server device 1 may be realized by different servers connected to each other via the communication network 2.

[0051] The series of processes performed by the device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. A computer program for realizing each function of the server device 1 according to this embodiment may be created and implemented in a PC or the like. A computer-readable recording medium in which such a computer program is stored may also be provided. The recording medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. The above computer program may also be distributed, for example, via a communication network 2 without using a recording medium.

[0052] Additionally, the processes described herein do not necessarily have to be performed in the order described, some process steps may be performed in parallel, additional process steps may be employed, and some process steps may be omitted.

[0053] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects. [Explanation of symbols]

[0054] 1. Server device 2. Communication Network 3. User terminal 101 CPU 102 Memory 103 Storage device 104 Communication Interface 105 Input Device 106 Output Device 111 Generation information acquisition unit 112 Result image generation unit 131 Generation information storage unit 132 Image information storage unit

Claims

1. A generation support device that supports generation of a result image, a generation information acquisition unit that acquires generation information including at least style information related to a style of the resultant image and an element image that constitutes a part of the resultant image from a user; a resultant image generation unit that inputs text information generated based on the generation information into a generation model and generates the resultant image based on output information output from the generation model; A generation assistance device comprising:

2. the generation information acquisition unit further acquires position information of the element images in the resultant image; The generation support device according to claim 1 .

3. The style information includes information of a web address, the generation information acquisition unit determines the style information based on web information included in the website specified by the web address as the style information; The generation support device according to claim 1 or 2,

4. the information acquisition unit accepts upload or selection of the element image, and acquires the position information based on an arrangement of the element image in a frame of the resultant image; The generation support device according to claim 2 .

5. the information acquisition unit acquires the generation information through input by the user in a chat format; The generation support device according to claim 1 or 2,

6. when acquiring the input in the chat format, the information acquisition unit presents to the user a suggestion of information required as the generation information; The generation support device according to claim 5 .

7. A generation assistance program for assisting in the generation of a resultant image, The processor: a generation information acquisition step of acquiring generation information including at least style information related to a style of the resultant image and an element image constituting a part of the resultant image from a user; an image generation step of inputting text information generated based on the generation information into a generation model, and generating the resultant image based on output information output from the generation model; A generation support program that executes the above.

8. A method for assisting in the generation of a result image, comprising: The processor: a generation information acquisition step of acquiring generation information including at least style information related to a style of the resultant image and an element image constituting a part of the resultant image from a user; an image generation step of inputting text information generated based on the generation information into a generation model, and generating the resultant image based on output information output from the generation model; A generation assistance method for executing the above.

Images