Generation support device, generation support program, and generation support method

Abstract

To provide a generation support device that supports the generation of a resultant image.SOLUTION: In an evaluation system in which a server device 1 is communicatively connected to a user terminal 3 via a communication network 2, the server device includes: a generation information acquisition unit that acquires, from a user, generation information including at least style information related to a style of a resultant image and an element image constituting a part of the resultant image; and a resultant image generation unit that inputs text information generated based on the generation information into a generation model and generates the resultant image based on output information output from the generation model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a generation support device, a generation support method, and a generation support program. [Background technology]

[0002] Recently, images have been generated by various methods. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7169027 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, Patent Document 1 proposes a technology for generating character images using machine learning. It is being done.

[0005] However, the technology in Patent Document 1 generates images of characters in any pose. This is the only thing they can do and they cannot be applied to various image generation.

[0006] The present invention has been made in view of the above background, and is directed to a method for enabling a user to easily obtain a desired image. The purpose is to generate [Means for solving the problem]

[0007] In order to solve the above problems, the present disclosure provides a generation support device that receives at least In addition, style information relating to the style of the resultant image and element images constituting a part of the resultant image are stored. a generation information acquisition unit for acquiring generation information including the image; and a text generated based on the generation information. inputting store information into a generative model, and based on output information output from the generative model, and a result image generating unit that generates a result image.

[0008] Other problems and solutions disclosed in this application are described in the embodiments and drawings. will be made clearer. Effect of the Invention

[0009] According to the present invention, a user can easily generate a desired image. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an example of the overall configuration of an evaluation system according to an embodiment of the present invention; [Diagram 2] 2 is a diagram illustrating an example of a hardware configuration of a server device 1 according to the embodiment. FIG. [Diagram 3] 2 is a diagram illustrating an example of a functional configuration of a server device 1 according to the embodiment. FIG. [Figure 4] 13 is a diagram showing an example of basic information stored in a generation information storage unit 131. FIG. [Diagram 5] 13 is an example of a screen on which the generation information acquisition unit 111 acquires position information of a partial image and a material image. [Figure 6] FIG. 11 is a diagram illustrating an example of a process of the server device 1 according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Summary of the Invention> [Item 1] A generation support device that supports generation of a result image, The user provides at least style information regarding the style of the result image and a generation information acquisition unit that acquires generation information including an element image that constitutes a part of the image; Text information generated based on the generation information is input to a generation model, and the generation model a resultant image generating unit that generates the resultant image based on output information output from the A generation assistance device comprising: [Item 2] The generation information acquisition unit further acquires position information of the element image in the resultant image. thing, 2. The generation assistance device according to item 1, [Item 3] The style information includes information of a web address, The generation information acquisition unit is configured to acquire information about a website included in the website specified by the web address. the style information determined based on the block information is set as the style information; 3. The generation assistance device according to item 1 or 2, [Item 4] The information acquisition unit accepts upload or selection of the element image, and acquiring the position information by arranging the element images in a frame; 3. The generation assistance device according to item 2, characterized in that [Item 5] The information acquisition unit acquires the generation information through input by the user in a chat format. To do, 3. The generation assistance device according to item 1 or 2, [Item 6] When acquiring an input in the chat format, the information acquisition unit acquires the generated information as providing said user with an indication of required information; 6. The generation assistance device according to item 5, [Item 7] A generation assistance program for assisting in the generation of a resultant image, The processor: The user provides at least style information regarding the style of the result image and a generation information acquisition step of acquiring generation information including an element image that constitutes a part of the image; Text information generated based on the generation information is input to a generation model, and the generation model an image generating step of generating the resultant image based on output information output from the A generation support program that executes the above. [Item 8] A method for assisting in the generation of a result image, comprising: The processor: The user provides at least style information regarding the style of the result image and a generation information acquisition step of acquiring generation information including an element image that constitutes a part of the image; Text information generated based on the generation information is input to a generation model, and the generation model an image generating step of generating the resultant image based on output information output from the A generation assistance method for executing the above.

[0012] FIG. 1 is a diagram showing an example of the overall configuration of an evaluation system according to an embodiment of the present invention. The morphology generation support system includes a server device 1. The server device 1 The terminal 3 is communicably connected via a communication network 2. The communication network 2 includes: For example, the Internet is a network that can be used over public telephone lines, mobile phone lines, wireless communication channels, and Ethernet. It is constructed using Net (registered trademark) and other technologies.

[0013] ==Server device 1== The server device 1 is, for example, a general-purpose computer such as a workstation or a personal computer. It may be a computer or it may be logically implemented by cloud computing. In this embodiment, for convenience of explanation, one unit is illustrated, but the number of units is not limited to this. The number of units is not limited, and multiple units may be used.

[0014] ==User terminal 3== The user terminal 3 is a computer operated by a user who generates an image. These include smartphones, tablet computers, and personal computers. For example, the server device 1 is accessed by an application or a web browser executed on the user terminal 3. can be accessed.

[0015] FIG. 2 is a diagram showing an example of the hardware configuration of the server device 1. The illustrated configuration is This is merely an example, and the server device 1 may have other configurations. Memory 102, storage device 103, communication interface 104, input device 105, output device The storage device 103 is, for example, a hardware device that stores various data and programs. These include hard disk drives, solid state drives, and flash memory. The interface 104 is an interface for connecting to the communication network 2, e.g. For example, an adapter for connecting to an Ethernet (registered trademark), modem for wireless communication, wireless communication device for wireless communication, USB (Universal Serial Bus) for serial communication Serial Serial Bus (SMA) connector and RS232C connector. The device 105 is a device for inputting data, such as a keyboard, a mouse, a touch panel, a button, The output device 106 outputs data, for example, a display. Each of the functional units of the server device 1 is a processor. 101 reads a program stored in a storage device 103 into a memory 102 and executes it. Each storage unit of the server device 1 is provided by a memory 102 and a storage device 103. It is implemented as part of the storage area provided.

[0016] FIG. 3 shows the functional configuration of the server device 1. As shown in FIG. 3, the server device 1 includes: A generation information storage unit 131, a result image information storage unit 132, and a generation information acquisition unit The image processing unit 110 includes a processing unit 111 and a result image generating unit 112.

[0017] The generation information storage unit 131 and the result image information storage unit 132 will be described below. .

[0018] The generation information storage unit 131 stores a result image (generated by the server device 1) as shown in FIG. The information used to generate the image (hereinafter, referred to as generation information) is stored. and style information (including text information, web addresses, web information, etc.) The generation information may include information on element images that are the basis for forming a part of the resultant image. The element image may include, for example, the subject of the result image (e.g., a person or object, which will be described below). When the generation information acquisition unit 111 generates text to be input to the generation model, the name of the image is A partial image is an image of a subject (subject to be described as the content of the application), If it is an image of an object, for example, an image of a product, an image containing a product, or an image of the appearance of a product's container or outer box, etc. The element images may also include source images that are not the subject of the result image. The generation information may include, for example, information on the position of a partial image and a material image in the result image. may be used, but is not limited to these.

[0019] The style is a style in the design of the result image generated by the image generation support device. For example, the requirements for the resulting image (such as the objects, people, scenery, etc. contained in the image), The concept (target, story, etc.), color, texture (images that can be felt by visual senses) texture of the image surface, layout (positioning of elements and their relative positions, etc.), font Aesthetic attributes and characteristics such as shape (sharp angles, rounded shapes, straight lines, curves, etc.) represents, but is not limited to,

[0020] The material images include images of hands, faces, plants, everyday items, tables, and geometric shapes such as circles, triangles, and squares. Free-form figures that are not bound by academic or geometric rules, and the resulting image The material image is an image that is the basis of a part of the background of the image to be generated. It may also include, but is not limited to, images (templates).

[0021] The resultant image information storage unit 132 stores the resultant image generated by the resultant image generating unit 112 .

[0022] The following describes each processing unit of the generated information acquisition unit 111 and the result image generation unit 112. do.

[0023] The generation information acquisition unit 111 receives, for example, the generation information from the user terminal 3 via the communication network 2. Then, style information on the style of the result image necessary for generating the result image and the result image are generated. A raw image including element images constituting an image and position information of the element images in the resultant image. The generation information acquisition unit 111 stores the acquired generation information in the generation information storage unit 13. The communication in the transmission and reception may be either wired or wireless. Any communication protocol may be used as long as communication can be performed.

[0024] The generation information acquisition unit 111 may acquire the generation information in the form of text information. 111 may obtain a sentence indicating a result image to be generated by a user's input operation, or The above words may be acquired. In addition, the generation information acquisition unit 111 may acquire the style of the result image to be generated. A sentence or word representing the file is presented to the user, and the sentence or word selected by the user is It may be acquired as generation information.

[0025] The generation information acquisition unit 111 acquires information on a web address (such as a URL) as the generation information. The generation information acquisition unit 111 may acquire the generated information included in the website specified by the web address. The web information contained in the web page can be acquired, the style can be determined, and the generated information can be used. The text information, image information, video information, code information (website information, etc.) contained in the website The code that makes up a website, such as HTML, CSS, Javascript t, etc., but is not limited to these. 111, for example, determines the style of the target, concept, etc. based on the text information. In this case, the generated information acquisition unit 111 may perform, for example, a morphological analysis of the text information. Based on the information on the words and their number, the style of the target, concept, etc. can be determined. However, the method is not limited to these. Even if the style, such as color, texture, font, shape, etc., is determined from information or code information, In this case, the generated information acquisition unit 111 analyzes the image information or video information and The style may be determined from the color, texture, font, shape, etc., contained in the code information. The color, texture, shape, and font information of the web background image are stored. The style may be determined based on, but is not limited to, these methods.

[0026] The generation information acquisition unit 111 acquires element images that are the basis for forming a part of the resultant image. The generation information acquisition unit 111 may accept uploading of element images. The acquisition unit 111, as shown in FIG. 5, for example, acquires a material image (for example, 201 in FIG. 5) from a server. The image data is stored in the server device 1 and presented to the user terminal 3, and the user selects the material image. The material image selected by the user may be acquired as generation information.

[0027] The generation information acquisition unit 111 acquires, for example, position information of element images in a resultant image. The position information indicates the coordinates of the element image in the result image, for example, a specific corner of the result image. The XY coordinates are based on a specific position such as the center of the element image. In this case, the generation information acquisition unit 111 may generate a result image as shown in FIG. A frame (e.g. 202) according to the shape of the frame may be horizontal, square, vertical, etc. However, the presenting information is not limited to these) to the user terminal 3. In the frame, information on the user's operation on the user terminal 3 is acquired, and the material image in the resultant image is In this case, the generation information acquisition unit 111 acquires position information of the image, for example, It accepts drag and drop operations, and allows placement of partial images (203) or material images (204). The generation information acquisition unit 111 acquires the position information of the element image. The element images may also be enlarged, reduced, rotated, inverted, or transformed. The information acquisition unit 111 acquires, as position information, the anteroposterior relationship of a plurality of element images (for example, layer information In addition, the position information may be information on the positional relationship between element images (part The information may be, for example, information indicating that the material image is located below the component image.

[0028] The generation information acquisition unit 111 may acquire the generation information in a chat format. The information acquisition unit 111 performs word binarization of text information acquired in a chat format by morphological analysis or the like. In this case, the generated information acquisition unit 111 Regarding the generated information obtained from users, Please tell us about some websites that might be useful to you. In this case, the generation information acquisition unit 111 may provide support to make it easier for the user to recognize the From a list of generated information required for generating an image, which has been prepared in advance, For information that is not available or that could not be determined from the generated information obtained, The method is not limited to providing guidance.

[0029] The generation information acquisition unit 111 acquires a process to be input to the image generation model based on the acquired generation information. prompts or prerequisite conditions, etc. (collectively referred to as prompt information in this specification) The prerequisites are, for example, the image size, frame shape, file size, resolution, etc. The generated information acquisition unit 111 may generate information such as the degree of the generated information, but is not limited to this. The prompt includes at least a text representing a style. For example, the prompt information is generated using a feature extraction module and a language model. The generation information acquisition unit 111 generates one or more pieces of prompt information, presents them to the user, and The selection or editing of the target may be accepted.

[0030] The generation information acquisition unit 111 receives the image generated by the result image generation unit 112 when generating the prompt. Depending on the type of generative model used for generation, the structure of the generated prompt may be changed. The information acquisition unit 111 may generate a sentence-type prompt, or a word-arrangement prompt. It may also generate prompts for, for example, enclosing important words in parentheses, order at the beginning of the prompt, including multiple important words, etc. By indicating the importance of each word, we can generate prompts that allow the generative model to recognize important words. It may be possible to do so.

[0031] The prompt generated by the generation information acquisition unit 111 includes at least text representing the style. The generation information acquisition unit 111 may generate a plurality of prompts. For example, the generated information acquisition unit 11 may have a certain degree of randomness in the text included in the generated information acquisition unit 11. 1 is the prompt that is selected based on the semantic distance and similarity between the style and the text. Specifically, the generation information acquisition unit 111: For example, style information related to the "sea" is included in the generated information acquired through user input, etc. If the word contains multiple words, it will prompt other words that are close in meaning to "sea" or have a high similarity to it. The result image generating unit 112, which will be described later, generates a prompt by including the By using prompts to generate a result image, the image is closer to the one the user wants to generate. Conversely, the generation information acquisition unit 111 can generate a result image based on a user's input, etc. When the generated information obtained from the Generate prompts by including other words that are less similar or less relevant to the prompt. The result image generating unit 112, which will be described later, uses these prompts to generate a result image. This allows the user to choose the style of the result image when they do not yet have an image of the sea. It is also possible to make it easier to consider the direction of the prompt. The randomness may have other effects as well.

[0032] The generation information acquisition unit 111 performs the following on the first resultant image generated by the resultant image generation unit 112: The generation information acquired by the generation information acquisition unit 111 may be additionally acquired. Used to modify or add to the prompt used when generating the first result image, The generating unit 112 uses this when generating the second resultant image.

[0033] The result image generating unit 112 generates, for example, style information, element images, and position information. The resultant image generating unit 112 generates a resultant image based on at least one of the above. Generation information based on at least one of the tile information, the element image, and the position information. The prompt information generated by the acquisition unit 111 is input to the generation model, and the image output by the generation model is The result image generating unit 112 also acquires the image output by the generative model as the result image. Alternatively, the output image may be processed to generate a resultant image. 112 presents the generated result image to the user. The user can download the presented image. It can be downloaded.

[0034] The generative model used by the resultant image generating unit 112 to generate the resultant image is implemented in the server device 1. The communication network 2 may be implemented in another server accessible via the communication network 2. For this reason, the generative model may be implemented in the server device 1. If so, the result image generating unit 112 inputs the prompt information into the generation model, If the driver is installed in another server, the result image generator 112 generates the prompt information. The generated information is transmitted to the generative model via a communication network 2. Entering prompt information into a Generative Model, including sending prompt information to the Generative Model It is expressed as ``to exert one's strength.''

[0035] The generative model may be, for example, a specific input vector or a random nodal The model can receive the image size and generate an image from that information. The generator converts input information into appropriate The generator converts the feature or pattern into an image. Convolutional Neural Network k,CNN,Transformer,or other derivations. Built using a learning architecture, but other architectures are available In addition, the generative model may include, for example, a discriminator. The classifier distinguishes whether an image is real or a fake image generated by the generator. The classifier is constructed using a network such as CNN, but is not limited to these. The generative model may, for example, comprise a genetic adversarial network (GAN). The network trains the generator to generate more realistic images, while the classifier learns to recognize the real It learns to improve its ability to distinguish between images of objects and fake images.

[0036] The result image generating unit 112 may generate two or more result images. 112 presents the generated result image to the user.

[0037] When the result image generating unit 112 presents the generated multiple result images to the user terminal 3, In the user terminal 3, the user selects an image that is close to the result image that is to be generated from multiple images, or The selection operation of the image that is not included in the selection result is accepted, and the result image selected by the selection operation is used as the basis. In addition, a result image may be generated. In this case, the result image generating unit 112 generates, for example, an image Based on the features of result image A, which is selected as the result image close to the one we want to generate, A resultant image B similar to the image A may be generated. is the process input to the generative model when generating the result image A selected by the selection operation. Modify the information in the prompt or reproduce an image similar to or a variation of the selected result image A We generate prompts that show the However, the present invention is not limited to this method.

[0038] The resultant image generating unit 112 performs generation information acquisition for the generated resultant image (first resultant image). When the acquisition unit 111 acquires additional information from the user, a second result image may be generated. In this case, the resultant image generating unit 112 uses the generative model when generating the first resultant image. The prompt information input to the user and the additional information are used to generate the prompt information. prompt information is input to a generative model, and based on the output information output by the generative model, All you have to do is generate the result image of step 2.

[0039] FIG. 6 is a diagram illustrating an example of the process of the generation support device according to the present embodiment.

[0040] The server device 1 acquires the generation information from the user (1001). The server device 1 generates a prompt based on the generated information (1002). The server device 1 receives output information of the generative model (result image ) is acquired (1004). The server device 1 presents the output information to the user (1005). .

[0041] Other examples are described below.

[0042] The server device 1 performs pre-processing of the partial image acquired by the generation information acquisition unit 111, for example. The server device 1 may, for example, determine the subject of the partial image and remove the background other than the subject portion. Also, the generation information acquisition unit 111 may emphasize a main subject from a partial image, for example. .

[0043] As a pre-processing step, the server device 1 performs, for example, a process of determining whether a subject included in a partial image is a subject or a color. The camera angle is detected by determining the positional relationship of the camera, and the generated information acquisition unit 1 11 may generate a prompt that, for example, Examples of the display angle include, but are not limited to, a display angle that specifies the angle at which the subject is displayed.

[0044] The generation information acquisition unit 111 performs generation information on the partial image after the above-mentioned preprocessing. Position information in the image may be obtained and a prompt may be generated.

[0045] The server device 1 may suggest styles to the user based on marketing information. The marketing information is information on the product that is the subject of the result image, etc., that was previously obtained, Information on the subject matter, such as information on the world, results of marketing surveys, etc., and information obtained from users It may also include information about the product's past sales performance, sales of similar products, etc. The device 1 may, for example, obtain information on the sales history of similar products for the product that is the subject of the image to be generated. Based on the information, the standard is determined based on sales websites and advertising images of similar products with high sales volume. In this case, the server device 1 recommends a similar product with a high sales volume to the user. The web address information of the website and the generated information acquisition unit 111 generate the style information. The user is prompted to enter text information to be included in the prompt (e.g., "luxury" or "natural"). This may be presented or may be included in the prompts used to generate the image.

[0046] The server device 1 stores a result image that a user has previously generated using the server device 1, or Based on the prompt information used to generate the result image, style recommendations are provided to the user. For example, the server device 1 may analyze result images generated by the user in the past, The style is determined based on the text information included in the prompt or the text, and the most frequently detected style is The style is presented to the user terminal 3, and the user is prompted to select whether or not to use the style in generating a result image. Specifically, the server device 1 may acquire, for example, the real-time data of the user in the past. If the system judges that only realistic images are being generated, the system will display the message "Do you want to generate realistic images? Yes A question such as "No" is presented to the user terminal 3 via chat or the like, and the user's selection operation is acquired. A prompt may be generated based on the answer selected by the user.

[0047] The server device 1 may generate information related to product sales in addition to images. The information generated by 1 is, for example, images for banner advertisements promoting products or campaigns, An effective catchphrase or copy that succinctly expresses the features of the product or brand. Product description, which is text information that describes the detailed explanation and features of the product, The design, layout, and product categories used on the top page of an EC site that sells Category page design, which is the design and display method of the page, specific campaign Design landing pages, social media and advertising platforms to highlight your products and This includes, but is not limited to, images and taglines for the website. When the server device 1 generates the above-mentioned information, the generated information acquisition unit 111 generates the information based on the acquired information. If the prompt is an image or design, the result image generator 112 generates the image. Enter a prompt, and if it is text information, use a text generation model (e.g., ChatGP Just input the prompt into a large language model (such as T) and generate it.

[0048] When acquiring an element image that constitutes a part of a result image, the server device 1 Images contained in the website specified by the web address may be obtained as element images. The server device 1 acquires all images contained in the website as element images and generates The image may be stored in the information storage unit 121, or may be generated from among the images included in the website. Accepts a user's selection operation regarding the image to be acquired as information, and displays the selected image They may be stored as element images.

[0049] The preferred embodiments of the present disclosure have been described above in detail with reference to the accompanying drawings. The technical scope of the disclosure is not limited to such examples. If a person makes such a claim, he / she may make various modifications within the scope of the technical idea described in the claims. It is apparent that various modifications and variations are possible, which are of course within the scope of the present disclosure. It is understood to be within the scope of the art.

[0050] The devices described in this specification may be implemented as a single device, or may be implemented in part or in whole as a single device. The part is realized by a plurality of devices (e.g., cloud servers) connected via a communication network 2. For example, the processor 101 and the storage device 103 of the server device 1 may may be realized by different servers connected to each other via a communication network 2.

[0051] The series of processes performed by the apparatus described in this specification may be implemented by software, hardware, and The present embodiment may be implemented using either software or a combination of software and hardware. A computer program for implementing each function of the server device 1 according to the embodiment is created, It is possible to implement such a computer program in a program such as C. A computer-readable recording medium may also be provided. Examples include magnetic disks, optical disks, magneto-optical disks, and flash memories. The computer program can be transmitted, for example, via a communication network 2 without using a recording medium. The content may be distributed in a format similar to that described above.

[0052] Additionally, the processes described herein do not necessarily have to be performed in the order described. Some processing steps may be performed in parallel. Alternatively, some processing steps may be omitted.

[0053] In addition, the effects described in this specification are merely illustrative or exemplary and are not limiting. In other words, the technology according to the present disclosure has the above-mentioned effects in addition to or in place of the above-mentioned effects. In addition, other effects that will be apparent to those skilled in the art from the description of this specification may be achieved. [Explanation of symbols]

[0054] 1. Server device 2. Communication Network 3. User terminal 101 CPU 102 Memory 103 Storage device 104 Communication Interface 105 Input Device 106 Output Device 111 Generation information acquisition unit 112 Result image generation unit 131 Generation information storage unit 132 Image information storage unit

Claims

1. A generation support device that supports generation of a result image, a generation information acquisition unit that acquires generation information including at least style information related to a style of the resultant image and an element image that constitutes a part of the resultant image; a resultant image generation unit that inputs text information generated based on the generation information into a generation model and generates the resultant image based on output information output from the generation model; A generation assistance device comprising:

2. The style excluding the layout of the element images, The generation support device according to claim 1 .

3. The style information includes information of a web address, the generation information acquisition unit determines the style determined based on web information included in the website specified by the web address as the style information. The generation support device according to claim 1 .

4. the generation information acquisition unit determines aesthetic characteristics based on web information including code information constituting the website as the style information; The generation support device according to claim 3 .

5. The style information includes information obtained by the generation information acquisition unit determining at least a concept of the resultant image from text information included in the website and at least aesthetic features of the resultant image from image information included in the website.

5. The generation support device according to claim 3.

6. The generation information acquisition unit suggests the style information based on marketing information including at least information about a product that is the subject of the result image or information about sales performance of a product similar to the product. The generation support device according to claim 1 .

7. The generation information acquisition unit suggests the style information based on information of a result image previously generated by the user or a prompt used when generating the result image. The generation support device according to claim 1 .

8. The result image generation unit presents a plurality of the result images to a user, accepts a selection operation by the user, and generates a further result image based on the result image selected by the selection operation. The generation support device according to claim 1 .

9. The generation information acquisition unit acquires the generation information through input by a user in a chat format. The generation support device according to claim 1 .

10. When the generation information acquisition unit acquires the input in the chat format, it presents to the user a suggestion of information required as the generation information. The generation support device according to claim 9.

11. A generation assistance program for assisting in the generation of a resultant image, The processor: a generation information acquisition step of acquiring generation information including at least style information related to a style of the resultant image and an element image constituting a part of the resultant image; a resultant image generating step of inputting text information generated based on the generation information into a generation model, and generating the resultant image based on output information output from the generation model; A generation support program that executes the above.

12. A method for assisting in the generation of a result image, comprising: The processor: a generation information acquisition step of acquiring generation information including at least style information related to a style of the resultant image and an element image constituting a part of the resultant image; a resultant image generating step of inputting text information generated based on the generation information into a generation model, and generating the resultant image based on output information output from the generation model; A generation assistance method for performing the above.

Images