IMAGE GENERATION DEVICE, IMAGE GENERATION METHOD, AND IMAGE GENERATION PROGRAM
The image generation device addresses inefficiencies in content generation by using a language model to derive metadata and combine image parts semantically, facilitating efficient and consistent image creation from text inputs.
Patent Information
- Application Number
- JP2023195717
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-11-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-11-17
AI Technical Summary
Existing content generation systems, particularly for image content, face inefficiencies due to the burdensome task of selecting and combining multiple parts, which can vary in workload and quality based on user knowledge, especially when generating large numbers of images.
An image generation device that uses a language model to derive metadata from text input and prompts, selecting and combining image parts semantically compatible with the text to generate images efficiently.
Enables the generation of image content with a simple procedure, reducing user workload and ensuring consistent quality by leveraging a language model to automate the selection and combination of image parts.
Smart Images

Figure 0007594075000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for generating image content from text information. [Background technology]
[0002] In recent years, with the widespread use of notebook PCs (personal computers), smartphones, tablet devices, and the like, users have more opportunities to use the Internet easily at any time and place. For example, users can use online web services via the Internet, such as shopping at EC (electronic commerce) sites and SNS (social network services). Companies providing such web services are trying to create more attractive web pages that appeal to users' visual senses in order to encourage more users to frequently use the web services they provide.
[0003] A web page is generally composed of one or more pieces of content (also called web content), each of which is composed of multiple parts. A web page creator creates a web page, for example, by generating one or more pieces of content and arranging each piece of content in a predetermined position. As a technology for generating content on a web page, Patent Document 1 discloses a content management device that generates content by combining multiple parts selected by a user from a storage unit in which multiple parts are stored and arranged in any position. This device makes it possible for a user to generate content by selecting and combining multiple parts. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-204184 A Summary of the Invention [Problem to be solved by the invention]
[0005] According to the technology disclosed in Patent Document 1, content can be generated by combining desired parts selected by a user from a plurality of parts pre-stored in a storage unit. However, the task of selecting a plurality of parts each time a piece of content is generated increases the load as the amount of content to be generated increases. Thus, by distributing the content generation task among a plurality of users, the workload per person can be reduced, but differences in work efficiency may occur depending on the prior knowledge of each user.
[0006] Such problems with work efficiency may become more pronounced when the content to be generated is image content consisting of multiple parts (image parts). For example, when generating a large number of image contents, the parts to be selected vary depending on the target image content, so the work load involved in selecting and generating the parts may be large. Furthermore, differences in prior knowledge of the types of parts that can be selected and the positions where parts can be placed among multiple users may result in large differences in work efficiency in generating image contents.
[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique for generating image content through a simple procedure. [Means for solving the problem]
[0008] In order to solve the above problem, one aspect of an image generation device according to the present invention includes an acquisition unit that acquires text information and a prompt, where the prompt is an instruction for outputting metadata including a plurality of attribute values, each of which is an attribute value for a plurality of attributes, so as to semantically match the text information; a derivation unit that derives metadata for the text information by inputting the text information and the prompt into a language model; and a generation unit that generates an image based on the derived metadata.
[0009] In order to solve the above problem, one aspect of an image generation method according to the present invention includes acquiring text information and a prompt, where the prompt is an instruction for outputting metadata including a plurality of attribute values, each of which is an attribute value for a plurality of attributes, so as to semantically match the text information, deriving metadata for the text information by inputting the text information and the prompt into a language model, and generating an image based on the derived metadata.
[0010] In order to solve the above problem, one aspect of the image generation program of the present invention is a program that causes a computer to execute an acquisition step of acquiring text information and a prompt, where the prompt is an instruction to output metadata including a plurality of attribute values consisting of attribute values for each of a plurality of attributes so as to semantically match the text information, a derivation step of deriving metadata for the text information by inputting the text information and the prompt into a language model, and a generation step of generating an image based on the derived metadata. Effect of the Invention
[0011] According to the present invention, it is possible to generate image content through simple procedures. The above-mentioned objects, aspects, and advantages of the present invention, as well as objects, aspects, and advantages of the present invention not described above, will be understood by those skilled in the art from the following detailed description of the invention by referring to the accompanying drawings and the claims. [Brief description of the drawings]
[0012] [Figure 1A] FIG. 1A is a conceptual diagram showing an input to an image generating device according to an embodiment and an output in response to the input. [Figure 1B] FIG. 1B is a conceptual diagram of the image content generation process. [Diagram 2] FIG. 2 illustrates an example of a functional configuration of an image generating apparatus according to an embodiment. [Diagram 3] FIG. 3 shows examples of a head part, a top part, and a bottom part. [Figure 4] FIG. 4 shows an example of file names and IDs given to parts. [Diagram 5] Figure 5 shows an example of the code for an SVG file. [Figure 6] Figure 6 shows an example of a screen containing a prompt and text information. [Figure 7] FIG. 7 shows an example of metadata derived from input text information. [Figure 8] FIG. 8 shows three selected parts from the metadata derived in FIG. [Figure 9] FIG. 9 shows an example of a joining procedure for the three parts shown in FIG. [Figure 10] FIG. 10 shows an example of a screen for correcting a generated image. [Figure 11A] FIG. 11A shows a conceptual diagram of an image in which the color of the dominant portion in the generated image has been changed. [Figure 11B] FIG. 11B is a conceptual diagram showing an image in which the colors of one or more partial regions in the generated image have been changed. [Figure 12] FIG. 12 shows a flow of processing executed by an image generating device according to an embodiment. [Figure 13] FIG. 13 shows a flow of processing executed by an image generating device according to a modified embodiment. [Figure 14] FIG. 14 shows an example of the hardware configuration of an image generating device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, with reference to the attached drawings, an embodiment for carrying out the present invention will be described in detail. Among the components disclosed below, those having the same functions are given the same reference numerals, and their description will be omitted. Note that the embodiment disclosed below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.
[0014] [Overview of the operation of the image generating device] First, an overview of the operation of an image generating device 10 according to this embodiment will be described with reference to Figs. 1A and 1B. Fig. 1A is a conceptual diagram showing an input to the image generating device 10 and an output in response to the input. The image generating device 10 may be configured as a general-purpose PC (personal computer) equipped with a display unit (display surface) such as a liquid crystal display and a communication unit. Note that the image generating device 10 is not limited to a general-purpose PC, and may be, for example, a mobile terminal such as a smartphone or a tablet PC. The image generating device 10 may also be equipped with a separate display unit.
[0015] User 1 directly inputs text information (hereinafter also simply referred to as text) representing a desired image content (hereinafter also simply referred to as image) by operating image generating device 10. Alternatively, user 1 may indirectly input text by operating user equipment (UE) 11 to transmit text to image generating device 10. The text representing the desired image may be configured to express or explain at least a part of the image. Furthermore, if user 1 cannot find appropriate text representing the desired image, the text to be input may include an indirect expression or metaphorical expression for at least a part of the image, such as "something similar to...".
[0016] The image generating device 10 selects a plurality of parts that semantically match the text (i.e., the text description) received directly from the user 1 or indirectly via the user device 11, and generates one image from the plurality of parts. Note that in this embodiment, the image generating device 10 is assumed to generate one image, but may be configured to generate multiple images from the selected plurality of parts. The image generating device 10 outputs the generated image to the user 1 as a response to the received text. As an example of output, the image generating device 10 may display the generated image on a display unit, or may transmit the generated image to the user device 11 via a communication unit.
[0017] The user device 11 is, for example, a device such as a smartphone or a tablet, and is configured to be able to communicate with the image generating device 10 via wired communication or wireless communication. The user device 11 has a display unit (display surface) such as a liquid crystal display, and the user 1 can perform various operations using a GUI (Graphic User Interface) provided on the display unit. The operations include various operations on content such as images displayed on the screen, such as tapping, sliding, and scrolling using a finger or a stylus. The user device 11 may be configured to display an image generated by the image generating device 10 on the display unit when the user device 11 receives the image from the image generating device 10. The user device 11 may also be provided with a separate display unit. In this way, the user 1 can receive from the image generating device 10 an image consisting of a plurality of parts that is semantically suited to the text, by the simple procedure of inputting the text representing the desired image.
[0018] FIG. 1B shows a conceptual diagram of image content generation processing by the image generating device 10. For explanation, FIG. 1A is referred to. As shown in FIG. 1B, first, the image generating device 10 acquires text 102 by input of the user 1. Furthermore, the image generating device 10 acquires a prompt 101. The prompt 101 is a command (instruction) for a language model, and includes a command for outputting metadata under a predetermined condition so as to be semantically compatible (matched) with the text 102. The image generating device 10 may acquire the prompt 101 input by the user 1, or may acquire the prompt 101 generated by the image generating device 10 or another device. The image generating device 10 applies (inputs) the acquired prompt 101 and text 102 to a trained language model.
[0019] Next, the image generating device 10 performs a metadata derivation process 103 using a language model from the prompt 101 and the text 102. In this embodiment, the language model refers to a Natural Language Processing (NLP) model. Known language models include large language models (LLMs) such as BERT (Bidirectional Encoder Representations from Transformers) and the GPT series (for example, ChatGPT). In this embodiment, an example is described in which an LLM that operates according to the command of the prompt is used. It is possible to use other language models. The language model according to this embodiment is configured to output (derive) metadata 104 that is semantically compatible with the text 102 according to the command of the prompt 101. Details of the metadata derivation process 103 using the language model will be described later.
[0020] Once the metadata 104 has been derived, the image generating device 10 performs image part selection (sampling) and combination processing 105 based on the metadata 104. Specifically, the image generating device 10 selects a plurality of image parts based on the metadata and combines the selected image parts. Image content 106 is generated by combining the image parts. The image part selection and combination processing 105 will be described in detail later. The image generating device 10 outputs the generated image content 106 to the user 1. The configuration and operation of the image generating device 10 will be described in detail below.
[0021] [Functional configuration and operation of the image generating device] The image generating device according to this embodiment is configured to use a language model to select from text a number of image parts that are semantically compatible with the text, and to generate an image (image content) by combining the selected image parts. In this embodiment, the image parts and the generated image are images in SVG (Scalable Vector Graphics) files. Since SVG records images in a text format (code written in XML) based on XML (Extensible Markup Language), it is also possible to edit the images with a text editor. In other words, SVG is an image that can also be edited with code (text).
[0022] FIG. 2 shows an example of the functional configuration of the image generating device 10 according to the present embodiment. As an example of the functional configuration, the image generating device 10 has a parts management unit 201, a naming unit 202, an ID (identifier) assignment unit 203, a text information acquisition unit 204, a prompt acquisition unit 205, a metadata derivation unit 206, an image generating unit 207, an output control unit 208, and a data storage unit 210. The data storage unit 210 is configured to be able to store a parts group 211, a language model 212, and an image content group 213. Note that the entire image generating device 10 may not be provided in one device, but may be provided in a plurality of devices. For example, a part of the image generating device 10 may be provided in an external server device. In this case, the following functions are realized by cooperation between the image generating device 10 and the external server device.
[0023] The parts management unit 201, naming unit 202, and ID assignment unit 203 function as a preprocessing unit that executes preprocessing. The preprocessing unit executes preprocessing including assigning file names and IDs to a plurality of image parts (hereinafter also simply referred to as parts) that can constitute an image. The file name is composed of a character string (word) that expresses the characteristics of the part, and the ID is composed of a character string that expresses the characteristics of a partial area (subpart) of the part.
[0024] The parts management unit 201 first acquires a large number of parts that can constitute the image content. The large number of parts are created in advance and stored, for example, in a predetermined storage unit in the image generating device 10 or in a device other than the image generating device 10. In this embodiment, the generated image is an illustrated content representing an entire human being, and each image can be generated by combining three types of parts. The three types are the head, the upper body, and the lower body. Thus, the parts are classified into three types of parts. The first type is a head part representing the head (Head), the second type is a top part representing the Top (upper body, body), and the third type is a bottom part representing the Bottom (lower body, legs). In this embodiment, an image is generated from three parts, but the parts may be created and classified so that an image is generated from two parts or four or more parts.
[0025] 3 shows examples of parts, including a head part group 30 which is a group of head parts, a top part group 31 which is a group of top parts, and a bottom part group 32 which is a group of bottom parts. At least a part of each part may be colored in a predetermined color.
[0026] Each head part in the head part group 30 represents a human head facing forward or sideways. The head parts may include heads with different hairstyles (including shaved heads) and different hair colors, as well as heads wearing headphones. The head parts may also include heads wearing any wearable object, such as glasses, hats, hijabs, bandanas, accessories, etc. The head parts may also include heads wearing the hood of clothing connected to the upper body, such as a hoodie.
[0027] Each top part in the top part group 31 represents an upper body part (from the neck to the torso part) facing forward or sideways. The top part is an upper body part wearing one or more clothes or fabrics such as a shirt, a suit, a tie, a cardigan, an apron, a swimsuit, a scarf, or a stole (hereinafter also referred to as upper clothes), or an upper body part wearing nothing. The top part may also be an upper body part wearing a part other than the hood part of a piece of clothing connected to the head, such as a parka. The top part may also include an arm part and / or a hand part. The arm part and / or the hand part may have a right part and / or a left part, respectively, and when having both left and right parts, the right part and the left part may be configured to take different poses or postures. The arm part and / or the hand part may also be configured to hold an object that can be held or grasped, such as a bag or a smartphone. The hand part may also be wearing gloves, and the nail part of the hand part may have nail art.
[0028] Each bottom part in the bottom part group 32 represents a leg (from the waist to the foot) facing forward or to the side (side). The bottom part is a lower body part wearing one or more pieces of clothing or cloth such as a skirt, pants, or swimsuit (hereinafter also referred to as lower clothes), or a lower body part wearing nothing. The bottom part may be configured to wear footwear such as shoes (including sneakers and pumps) or sandals. The bottom part may not represent a standing state. For example, the bottom part may be configured to sit on a chair such as a sofa or an office chair. The bottom part does not necessarily have to have both a right foot and a left foot, and if it has both left and right feet, the right foot and the left foot may not be the same length. The bottom part may be represented so as to be associated with an object in a part from the waist to the foot. For example, the bottom part may be configured to include a ball near the right foot or the left foot to represent a foot kicking a ball.
[0029] The sizes of the head parts do not have to be the same. The same applies to the top parts and bottom parts. For example, a top part including an upper body wearing a long apron or a dress can be relatively long vertically compared to a top part including an upper body wearing a short shirt. Also, a bottom part including a lower body sitting sideways on a chair can be relatively long horizontally compared to a bottom part including a lower body standing in front. However, since the head part, the top part, and the bottom part are ultimately combined to generate one image, each part is configured so that the combined image does not become unnatural. For example, the areas of the neck part and the face part are configured so that the neck part of the top part is not wider horizontally than the face part of the head part when the head part and the top part are centered and combined. Also, in order to combine the head part, the top part, and the bottom part, a predetermined line can be set in advance for each part, as described later with reference to FIG. 9.
[0030] The naming unit 202 assigns a file name expressing the characteristics of each part acquired by the parts management unit 201 (close to the characteristics). In this embodiment, the file name is composed of a concatenation of multiple character strings expressing multiple characteristics included in the part. The naming unit 202 may assign a file name to the part by an input operation by the user 1. For example, the user 1 looks at the part displayed on the display unit of the user device 11 or the image generation device 10, visually grasps (recognizes) multiple characteristics of the part, and directly or indirectly inputs character strings expressing the multiple characteristics to the image generation device 10. Then, the parts management unit 201 can generate a file name by concatenating the input character strings, and assign the file name to the part. Alternatively, the naming unit 202 may assign a file name to the part according to a predetermined rule using image recognition processing. For example, the naming unit 202 extracts multiple characteristics of the part by performing image recognition processing (including object recognition and face recognition) on the part, and obtains (generates) character strings expressing the characteristics of the multiple parts by referring to the characteristics to a predetermined rule. Then, the parts management unit 201 can generate a file name by concatenating the character strings and assign the file name to the part. The image recognition process may be a process using a trained machine learning model. In addition, when it is difficult to express the characteristics of a part, such as a hairstyle (including a shaved head) or a face shape of a head part in an identifiable manner using a character string, the part may be identified using a number according to a predetermined rule.
[0031] In this embodiment, the character strings selectable as character strings expressing the features of the parts are predefined for the head part, the top part, and the bottom part. The predefined character strings for the head part, the top part, and the bottom part are the head class, the top part class, and the bottom part class. The top part class is further classified into two classes, an upper body activity class and an upper clothing class, and the bottom part class is further classified into two classes, a lower body activity class and a lower clothing class. By an input operation or an image recognition process by the user 1, a plurality of character strings are selected for the head part from the character strings included in the head class. Similarly, a plurality of character strings are selected for the top part from the character strings included in the top part class, and a plurality of character strings are selected for the bottom part from the character strings included in the bottom part class.
[0032] Examples of file names given to parts are shown in Fig. 4. As described above, in this embodiment, parts are configured with SVG files, so the extension is svg. Fig. 4 shows a file name 401 for a head part 400, a file name 411 for a top part 410, and a file name 421 for a bottom part 420.
[0033] When the head part 400 is recognized as the head of a man wearing headphones facing sideways, its characteristics are expressed by three character strings, "Man (male)", "HeadPhone (headphones)", and "HeadSide14 (head side type 14)". The naming unit 202 can give the file name 401 "Man_HeadPhone_HeadSide14" to the head part 400 by concatenating the three character strings.
[0034] When top part 410 is recognized as facing forward and holding a barcode reader in hand, its characteristics are expressed by four character strings, "Front," "Holding," "BarCodeReader," and "CollarShirt." Naming unit 202 can give file name 411, "Front_Holding_BarCodeReader_CollarShirt," to top part 410 by concatenating the four character strings.
[0035] When bottom part 420 is recognized as standing facing forward and wearing a line skirt, its features are expressed by three character strings, "LegsFront," "Stand," and "AlineSkirt." Naming unit 202 can give file name 421 "LegsFront_Stand_AlineSkirt" to bottom part 420 by concatenating the three character strings.
[0036] The file name shown in Fig. 4 is an example, and for example, a file name different from the file name shown in Fig. 4 may be given to the head part 400. For example, the character strings representing the features of the parts may be different when input by a different user or when generated by a different image recognition process. In the file name shown in Fig. 4, multiple character strings are linked with an underscore, but the form in which the multiple character strings are linked is not limited to this.
[0037] The ID assigning unit 203 assigns IDs (identifiers) expressing the characteristics of a plurality of partial regions (subparts) in each part acquired by the parts management unit 201. Each part is composed of a plurality of partial regions. For example, the head parts included in the head parts group 30 in FIG. 3 are composed of at least two partial regions, namely, hair and skin. The ID assigning unit 203 assigns a character string expressing a characteristic, such as the name of the partial region, to each partial region as an ID. The ID assigning unit 203 may assign IDs to a plurality of partial regions by an input operation by the user 1. For example, the user 1 looks at a part displayed on the display unit of the user device 11 or the image generating device 10, visually grasps (recognizes) the characteristics of the plurality of partial regions included in the part, and directly or indirectly inputs a character string expressing the plurality of characteristics to the image generating device 10. Then, the ID assigning unit 203 can assign the input character string to the part as an ID. Alternatively, the ID assigning unit 203 may detect a plurality of partial regions included in the part by performing a known object detection process (image recognition process) on the part, and generate a character string. For example, object detection processing based on machine learning, such as the YOLO (You Only Look Once) algorithm, can be used. The ID assignment unit 203 can assign the generated character string to the part as an ID. For a partial area identified by the ID, it is possible to change the fill-in characteristics, such as color, in the finally generated image.
[0038] Furthermore, the ID assigning unit 203 associates multiple IDs for each part with the image file. In this embodiment, as described above, each part is configured with an SVG file. The ID assigning unit 203 associates an ID with a path element that draws each partial area in the SVG file.
[0039] An example of a plurality of IDs assigned to parts is shown in Fig. 4. In Fig. 4, an ID 402 for a head part 400, an ID 412 for a top part 410, and an ID 422 for a bottom part 420 are shown. The head part 40 is given three IDs, "Hair," "Skin," and "Headphone," according to the corresponding partial regions. The top part 41 is assigned three IDs, "Skin," "Shirt," and "BarCodeReader," according to the corresponding partial regions. The bottom part 42 is assigned three IDs, "Skirt," "Skin," and "Shoes," according to the corresponding partial region.
[0040] Furthermore, the ID assigning unit 203 associates the assigned ID with a path element in the SVG file. An example of code for an SVG file is shown in Fig. 5. In Fig. 5, a file name 501, an ID 502, and an SVG file 502 for a head part 500, and a file name 511, an ID 512, and an SVG file 513 for a head part 510 are shown.
[0041] The head part 500 is given a file name 501 of "Man_Bald_HeadsFront31" by the naming unit 202, and two IDs 502 of "Skin" and "Glasses" are given according to the corresponding partial area. In this case, the ID giving unit 203 creates an SVG file 503 by adding a path element ("<path··· / > "). Specifically, the ID assigning unit 203 associates (describes) "skin" with the id attribute of the path element that draws the partial area to which "Skin" is assigned, in the SVG file 503, and associates "glasses" with the path element of the area to which the partial area to which "Glasses" is assigned is drawn.
[0042] Further, the head part 510 is given a file name 511 of "Woman_MidumLengthHair_HeadsFront6" by the naming unit 202, and two IDs 512 of "Hair" and "Skin" are given according to the corresponding partial area. In this case, the ID assignment unit 203 associates the ID with the id attribute of the path element that draws each partial area in the SVG file 513. Specifically, the ID assignment unit 203 associates (describes) "hair" with the id attribute of the path element that draws the partial area to which "Hair" is assigned, and associates "skin" with the path element of the area that draws the partial area to which "Skin" is assigned, in the SVG file 513.
[0043] In addition, in the SVG file, the svg element "<svg···> The display area is set by the width and height attributes of the rect element "<rect··· / > The width and height attributes in the " specify a rectangular area, and the fill attribute specifies the color. In the path element, the d attribute specifies a partial area, and the fill attribute specifies the color. To specify the color in the fill attribute, you can use a hexadecimal color code that follows certain rules.
[0044] Furthermore, the ID assigning unit 203 can designate one or more partial regions occupying a large area as a dominant part among a plurality of partial regions included in each part. For example, the ID assigning unit 203 can designate a portion combining a hair part (excluding a shaved head) of a head part and an upper clothing part of a top part as a dominant part. When a dominant part is designated, the ID assigning unit 203 can associate the fact that the part is a dominant part with an id attribute of a path element that draws each partial region included in the dominant part in the SVG file.
[0045] By the file names given by the naming unit 202, the parts management unit 201 can classify and manage the acquired parts into a head parts group, a top parts group, and a bottom parts group. For example, the parts management unit 201 can classify and manage each part into a head parts group, a top parts group, and a bottom parts group by referring to the character strings of the head class, the top parts class, and the bottom parts class. When the naming unit 202 gives a file name to each part and the ID giving unit 203 gives an ID, the parts management unit 201 creates an SVG file for each part and saves it in the parts group 111. The process up to this point corresponds to pre-processing. When pre-processing is completed and the parts, each of which has a file name and one or more IDs assigned, are stored in parts group 211, image generating device 10 is ready to generate an image from text.
[0046] After completion of the preprocessing, the text acquisition unit 204 acquires text (text information) input by the user 1. The text is text that represents the characteristics of an image desired by the user 1. The image generating device 10 may acquire the text through an input operation by the user 1, or may acquire the text from the user terminal 11 operated by the user 1. Alternatively, the text acquisition unit 204 may acquire text input from a predetermined external device in response to an instruction from the user 1. The text acquisition unit 204 outputs the acquired text to the metadata derivation unit 206.
[0047] The prompt acquisition unit 205 acquires a prompt. The prompt is a command (instruction) for the language model 212, and is a command for outputting metadata under a predetermined condition so as to be semantically compatible (matched) with the text. The prompt acquisition unit 205 outputs the acquired prompt to the metadata derivation unit 206. The prompt acquisition unit 205 may acquire a prompt by an input operation by the user 1, or may acquire a prompt that is generated and stored in advance in the image generating device 10 or an external device. The language model 212 is an LLM configured to infer and output (derive) metadata that is semantically compatible with the description of the input text according to the command of the prompt.
[0048] The metadata derivation unit 206 can display the acquired prompt and text on the display unit of the image generating device 10. Fig. 6 shows an example of a screen 60 including a prompt and text, displayed on the display unit of the image generating device 10. On the screen 60, a prompt 62 acquired by the prompt acquisition unit 205 is displayed in a prompt box 61, and text 66 entered by the user 1 is displayed in a text box 65. The contents of the prompt box 61 and the text box 65 on the screen 60 can be edited by the user 1.
[0049] The description of the prompt 62 will be described. In this embodiment, the metadata derived by the metadata derivation unit 206 is metadata described in JSON format. Metadata in JSON format is composed of a pair of a key (also called an attribute) and an attribute value (json value) for the key. The key is a character string, and the value can be any of the following types: character string, numeric value, boolean value (true, false), array, object, and null. In this embodiment, the prompt 62 includes an instruction to select one of multiple selectable attribute values for multiple keys so that the value semantically matches the input text.
[0050] In the example of Fig. 6, the prompt 62 is a command statement for generating metadata in JSON format by selecting an attribute value that semantically matches the text for each of a number of set keys. The prompt 62 commands the user to enter a message into JSON with six keys, "gender", "who", "top_body_activity", "bottom_body_activity", "top attire", and "bottom attire". The prompt 62 also includes instructions to select an attribute value from a list at the bottom of the prompt 62, to wear clothing appropriate for the user's gender, and to select a value from an element of the list without changing the name of the value.
[0051] In the prompt 62, a plurality of selectable lists are described for each of the six keys. The elements "Man" and "Woman" are set as attribute values for the "gender" key. Meanwhile, the attribute values for the keys "who", "top_body_activity", "bottom_body_activity", "top attire", and "bottom attire" are set as character strings included in the head class, upper body activity class, upper clothing class, lower body activity class, and lower clothing class, respectively. Since there are many character strings included in the head class, upper body activity class, upper clothing class, lower body activity class, and lower clothing class, the detailed element lists of these attribute values are omitted in FIG. 6.
[0052] As described above, the head part, the top part, and the bottom part are each configured to face forward or sideways, and on the screen 60, the user 1 can select the front button 63 or the side button 64. On the screen 60, the front button 63 is selected. When the front button 63 is selected, the metadata derivation unit 206 can set a condition so that an attribute value marked with "Front" is selected in the metadata to be derived. Also, when the sideways button 64 is selected, the metadata derivation unit 206 can set a condition so that an attribute value marked with "Side" is selected in the metadata to be derived.
[0053] Although six keys are specified in the prompt 62 in Fig. 6, the number of keys is not limited to a specific number. For example, in the screen 60, the selection of the front button 63 or the side button 64 is conditioned to select an attribute value with "Front" or "Side" in the derived metadata, but a key (for example, a "Facing" key) that allows selection of either "front facing" or "sideways" as the attribute value may be set.
[0054] Moreover, the text 66 is text information expressing a desired image of the user 1, and as an example, "A woman wearing formal wear" is input. In the text 66 shown in FIG. 6, gender is specified, but there are no particular rules for the text to be input, and it is sufficient that the text (descriptive information) expresses (describes) the desired image (in this embodiment, an image of a person). The text may also include an indirect expression or metaphorical expression for at least a part of the image, such as "something similar to...".
[0055] After the prompt 62 and text 66 are input, the user 1 activates (selects) the execute button 67 on the screen 60, which causes the metadata derivation unit 206 to input the prompt 62 and text 66 to the language model 212. Note that the trigger for the metadata derivation unit 206 to input the prompt 62 and text 66 to the language model 212 to derive metadata is not limited to the activation of the execute button 67, and may be automatically triggered in response to completion of input.
[0056] The metadata derivation unit 206 inputs the prompt 62 and the text 66 into the language model 212, and derives (acquires) metadata in a JSON format according to the prompt 62 by inferring it. That is, the metadata derivation unit 206 causes the language model 212 to execute a metadata derivation process (inference process) from the prompt 62 and the text 66. FIG. 7 shows an example of a screen 70 including metadata 71 derived from the input prompt 62 and text 66. The prompt 62 is omitted in the screen 70. The metadata 71 describes attribute values selected for each of six keys, "gender", "who", "top_body_activity", "bottom_body_activity", "top attire", and "bottom attire", which are set in the prompt 62 shown in FIG. 6.
[0057] Specifically, the attribute values "Woman", "Woman_LongHair_HeadsFront3", "BodyFront_SpeakingThinking", "LegsFront_Stand", "SuitWoman", and "PencilSkirt" are selected for each of the six keys "gender", "who", "top_body_activity", "bottom_body_activity", "top attire", and "bottom attire" in the metadata 71. Since the front button 63 is selected on the screen 60 in Fig. 6, the attribute values including "Stand", namely "Woman_LongHair_HeadsFront3", "BodyFront_SpeakingThinking", and "LegsFront_Stand", are selected.
[0058] The derived metadata 71 may be edited by the user 1. For example, attribute values included in the metadata 71 may be modified. In addition, when the execute button 67 is activated again by the user 1 after the metadata derivation unit 206 acquires the metadata 71, the metadata derivation unit 206 may execute a metadata derivation process using the language model 212 from the same input data so as to acquire metadata including at least some attribute values different from the metadata 71. For example, this can be realized by the metadata derivation unit 206 changing any parameter of the language model 212 every time the execute button 67 is activated. In addition, when the execute button 63 is activated by the user 1, multiple sets of metadata may be derived, for example, by the metadata derivation unit 206 changing any parameter of the language model 212 in a timely manner. The metadata derivation unit 206 outputs the derived metadata to the image generation unit 207 .
[0059] The image generating unit 207 generates one image content by selecting (sampling) three parts from the parts group 211 based on the metadata derived by the metadata derivation unit 206, and combining the three parts. Note that when multiple sets of metadata are derived, the image generating unit 207 may generate image contents for at least the number of sets according to the metadata of each set.
[0060] FIG. 8 exemplarily shows three parts 80 selected based on the metadata 71 shown in FIG. 7. The image generating unit 207 references the six attribute values included in the metadata 71 to the file names of the parts group 211 stored in the preprocessing, and selects an SVG file having a file name having a character string that matches or is similar to the six attribute values (character strings of the attribute values). In this embodiment, the image generating unit 207 selects three types of parts: a head part 81, a top part 82, and a bottom part 83. The head part 81, the top part 82, and the bottom part 83 each have a file name having a character string that matches or is similar to the character strings of the six attribute values included in the metadata 71 (not shown), and are parts that semantically match (match) the text 66 ("woman wearing formal wear").
[0061] The image generating unit 207 generates one image by combining the three selected parts. The part combining procedure will be described with reference to FIG. 9. FIG. 9 is a diagram for explaining an example of the part combining procedure. For the purpose of explanation, an X axis and a Y axis that are perpendicular to each other are set as shown in the figure. The origin is the lower left vertex (corner) of the square of each of the head part 81, the top part 82, and the bottom part 83. When combining the head part 81, the top part 82, and the bottom part 83 selected by the image generating unit 207, a horizontal line (a line having the same Y coordinate) that is set in advance for each of the parts is used. A first line 91 is set for the head part 81 at the boundary between the face and the neck (i.e., the jaw line). A second line 92 that should overlap with the first line 92 in the head part 82 is set for the top part 82. A third line 93 is set for the top part 82 at the hem line of the upper clothes. A fourth line 94 is set in the bottom part 83 so as to overlap with the third line 93 in the top part 82 .
[0062] The image generating unit 207 centers the head part 81, the top part 82, and the bottom part 83. Then, the image generating unit 207 places the bottom part 83 at the backmost position, and overlaps the bottom part 83 so that the fourth line 94 of the bottom part 83 overlaps with the third line 93 of the top part 82. Furthermore, the image generating unit 207 overlaps the top part 81 so that the second line 92 of the top part 82 overlaps with the first line 91 of the head part 81. In this way, an image 95 is generated in which the head part 81, the top part 82, and the bottom part 83 are combined.
[0063] In the example of FIG. 9, the image 95 is generated by overlapping the bottom part 83, the top part 82, and the head part 81 in this order, but the image generating unit 207 may overlap two parts so that at least a part of one part overlaps the other part. For example, when the top part includes a pose in which the hands are folded in front so as to overlap the lower clothes, the image generating unit 207 may overlap the top part and the bottom part so that the part of the hand appears in front of the top part. In this case, a partial area of the top part that should appear in front of the bottom part may be set in the part in advance so that the partial area appears in front of the bottom part when overlapping with the bottom part. In addition, the procedure for combining the three parts is not limited to the procedure described with reference to FIG. 9, and the image generating unit 207 may be configured to combine the three parts using arbitrarily set coordinates and parameters.
[0064] The image generated by the image generating unit 207 is output by the output control unit 208. For example, the output control unit 208 may display the generated image on the display unit of the image generating device 10, or may transmit the generated image to the user device 11 via the communication unit. The output control unit 208 may store the generated image in the image content group 213 in association with the text acquired by the text information acquiring unit 204. For example, the image generating unit 207 may store the text 66 in FIG. 6 in association with the image 95 in FIG. 9 in the image content group 213. After the image 95 is generated, when the text information acquiring unit 204 acquires the text 66, the output control unit 208 can extract and output the image 95. This allows the output control unit 208 to output the image 95 from the same text 66 again, without the metadata derivation unit 206 and the image generating unit 207 performing metadata derivation, part selection, and combination, respectively.
[0065] The image generated by the image generating unit 207 may be modified by additional text input by the user. FIG. 10 shows an example of a screen 1000 when modifying a generated image. After the image 95 shown in FIG. 9 is generated, if the user 1 inputs text 1001 "change her hair style" in the text box 65 and activates the execute button 67, the metadata derivation unit 206 derives metadata 1002 in which the attribute value of the key "Who" is modified in the metadata 71. Compared with the metadata 71, the attribute value of the key "who" in the metadata 1002 has been changed from "Woman_LongHair_HeadsFront3" to "Woman_LongHair_HeadsFront20". The image generating unit 207 generates an image 1003 from the metadata 1002. Compared with the image 95 shown in FIG. 9, the hairstyle of the image 1003 has been changed.
[0066] It is also possible to change the color (or fill) of at least one of the multiple partial regions in the image generated by the image generating unit 207. Changing the color of a partial region of an image will be described with reference to Figs. 11A and 11B.
[0067] FIG. 11A shows a conceptual diagram of an image in which the color of the dominant part set by the ID assignment unit 203 has been changed in the generated image. In this example, the dominant part is the hair part of the top part and the upper clothing part of the top part. Image 1100 is an image generated by the image generation unit 207. When the image generation unit 207 receives an instruction from the user 1 including the designation of the dominant part and the designation of a predetermined color, it changes the color of the dominant part to the designated color. Images 1101, 1102, and 1103 are conceptual diagrams of images in which the colors of the hair part and the upper clothing part as the dominant parts have been changed from image 1100.
[0068] FIG. 11B shows a conceptual diagram of an image in which the color of one or more partial regions in a generated image has been changed. In this example, an image in which the skin and hair colors have been changed is shown. When the image generating unit 207 receives an instruction from the user 1 including the designation of a partial region and a designation of a predetermined color, it changes the color of the designated partial region to the designated color. Image 1104 is a conceptual diagram of an image generated when an instruction to change the color of the hair portion has been given, where the color of the hair portion has been changed from image 1101. Image 1105 is a conceptual diagram of an image generated when an instruction to change the color of the skin portion has been given, where the color of the skin portion has been changed from image 1101. Note that, although an example has been described here in which the color of at least a portion of the generated image is changed by an operation by user 1, image generation unit 207 may be configured to change the color in accordance with a predetermined program.
[0069] In this way, the image generating device 10 according to the present embodiment selects multiple parts that are semantically compatible with the text expressing an image desired by the user 1, and generates one image content by combining the selected parts. This makes it possible to generate image content that is semantically compatible with the text by the simple procedure of inputting the text.
[0070] FIG. 12 shows a flow of processing executed by the image generating device 10 according to this embodiment. This processing is performed after the above-mentioned preprocessing is completed. First, the text information acquiring unit 204 acquires text, and the prompt acquiring unit 205 acquires a prompt (S10). The metadata derivation unit 206 infers and derives metadata that is semantically compatible with the text by inputting the text and the prompt acquired in S10 into the language model 212 (S20). Next, the image generating unit 207 selects (samples) a plurality of parts based on the metadata derived in S20 (S30). In this embodiment, the image generating unit 207 selects three parts, a head part, a top part, and a bottom part, from the part group 211 based on the derived metadata. Then, the image generating unit 207 combines the selected parts to generate one image content (S40). The output control unit 208 outputs the generated image content (S50).
[0071] As an optional process, the text information acquisition unit 204 may modify the text acquired in S10 by a user operation in order to modify the metadata derived in S20 (S60). Alternatively, the text information acquisition unit 204 may acquire, as the modified text, a text other than the text acquired in S10 by a user operation. When the text is modified, the metadata derivation unit 206 derives metadata again from the modified text and the prompt acquired in S10 (S20). In addition, the text information acquisition unit 204 may modify the text acquired in S10 by a user operation after the image content is generated (S40) (S80). Furthermore, the metadata derivation unit 206 may modify the metadata derived in S20 by a user operation (S70). When the metadata is modified, the image generation unit 207 selects three parts, a head part, a top part, and a bottom part, from the parts group 211 based on the modified metadata (S30), and combines the selected parts to generate one image content (S40). After the output control unit 208 outputs the image content, the image generation unit 207 may change a partial region of the image content in response to a user operation (S90). For example, as described with reference to Figures 11A and 11B, the image generation unit 207 can change the color of a dominant part or the color of one or more partial regions in response to a user operation. In this case, the output control unit 208 outputs the changed image content (S50).
[0072] Also, in S20, for example, the metadata derivation unit 206 may derive multiple sets of metadata by timely changing any parameter of the language model 212. In this case, the image generation unit 207 may generate image contents for at least the number of sets according to the metadata of each set (S30, S40).
[0073] In this embodiment, the language model 212 requires the input of a prompt, but if the language model 212 is configured to operate in accordance with an instruction shown in the prompt 62 in FIG. 6 in advance, only text information may be input. FIG. 13 shows a process flow according to a modified example of this embodiment. In FIG. 13, the processes of S10 and S20 in FIG. 12 are changed to S11 and S21. In S11, the text information acquisition unit 204 acquires text from the user. In S21, the metadata derivation unit 206 derives metadata by inputting the text acquired in S11 to the language model 212. The subsequent processes are the same as those in FIG. 12.
[0074] In this embodiment, the purpose is to generate an illustration of a person, but the present embodiment can be applied to any image that can be generated by combining multiple parts. In this embodiment, the image file format of each part is assumed to be SVG, but images in other image file formats such as JPEG (Joint Photographic Experts Group) and PNG (Portable Network Graphics) may be used. In this embodiment, the metadata is assumed to be in JSON format, but is not limited to a specific format as long as it includes a description corresponding to a key (attribute) and an attribute value.
[0075] [Hardware configuration of image generation device] Next, a description will be given of an example of the hardware configuration of the image generating device 10. Fig. 14 is a block diagram showing an example of the hardware configuration of the image generating device 10 according to this embodiment. The image generation device 10 according to the present embodiment may be implemented on any single or multiple computers, mobile devices, or any other processing platform. 2, the image generating device 10 is illustrated as being implemented in a single computer, but the image generating device 10 according to the present embodiment may be implemented in a computer system including multiple computers. The multiple computers may be connected to each other via a wired or wireless network so as to be able to communicate with each other.
[0076] 12, the image generating device 10 may include a CPU (Central Processing Unit) 1401, a ROM (Read Only Memory) 1402, a RAM (Random Access Memory) 1403, a HDD (Hard Disk Drive) 1404, an input unit 1405, a display unit 1406, a communication I / F (communication unit) (interface) 1407, a GPU (Graphics Processing Unit) 1408, and a system bus 1409. The image generating device 10 may also include an external memory. The CPU 1401 generally controls the operation of the image generating device 10, and controls each of the components (1402 to 1408) via a system bus 1409, which is a data transmission path.
[0077] The ROM 1402 is a non-volatile memory that stores a control program and the like necessary for the CPU 1401 to execute processing. The program includes instructions (codes) for executing the processing according to the above-described embodiment. The program may be stored in a non-volatile memory such as the HDD 1404 or an SSD (Solid State Drive) or an external memory such as a removable storage medium (not shown). The RAM 1403 is a volatile memory and functions as a main memory, a work area, etc. of the CPU 1401. That is, when executing a process, the CPU 1401 loads necessary programs, etc. from the ROM 1402 into the RAM 1403 and executes the programs, etc. to realize various functional operations. The RAM 1403 may include the data storage unit 210 shown in FIG.
[0078] The HDD 1404 stores, for example, various data and various information required when the CPU 1401 performs processing using a program. The HDD 1404 also stores, for example, various data and various information obtained when the CPU 1401 performs processing using a program. The input unit 1405 is composed of a keyboard and a pointing device such as a mouse. The display unit 1406 is configured with a monitor such as a liquid crystal display (LCD). The display unit 1406 may function as a GUI (Graphical User Interface) by being configured in combination with the input unit 1405. The above-described input operation by the user can be performed via the input unit 1405 or the GUI.
[0079] The communication I / F 1407 is an interface that controls communication between the image generating device 10 and an external device. The communication I / F 1407 provides an interface with a network and executes communication with an external device via the network. Various data, various parameters, etc. are transmitted and received between the external device and the communication I / F 1407. In this embodiment, the communication I / F 1407 may execute communication via a wired LAN (Local Area Network) or a dedicated line conforming to a communication standard such as Ethernet (registered trademark). However, the network available in this embodiment is not limited to this, and may be configured as a wireless network. This wireless network includes wireless PANs (Personal Area Networks) such as Bluetooth (registered trademark), ZigBee (registered trademark), and UWB (Ultra Wide Band). It also includes wireless LANs (Local Area Networks) such as Wi-Fi (Wireless Fidelity) (registered trademark), and wireless MANs (Metropolitan Area Networks) such as WiMAX (registered trademark). It also includes wireless WANs (Wide Area Networks) such as 4G and 5G. Note that the network only needs to be able to connect the devices to each other so that they can communicate with each other, and the communication standard, scale, and configuration are not limited to those described above. The GPU 1408 is a processor specialized for image processing, and can perform predetermined processes in cooperation with the CPU 1401.
[0080] At least some of the functions of each element of image generating device 10 shown in Fig. 2 can be realized by CPU 1401 executing a program. However, at least some of the functions of each element of image generating device 10 shown in Fig. 2 may be operated as dedicated hardware. In this case, the dedicated hardware operates under the control of CPU 1401.
[0081] The disclosure of this embodiment includes the following configuration. [1] An acquisition unit that acquires text information and a prompt, the prompt being an instruction for outputting metadata including a plurality of attribute values each consisting of an attribute value for each of a plurality of attributes so as to semantically match the text information; a derivation unit that derives metadata for the text information by inputting the text information and the prompt into a language model; and a generation unit that generates an image based on the derived metadata. An image generating device having the following:
[0082] [2] The image generating device described in [1], further comprising a memory unit that stores a plurality of image parts, wherein the generation unit selects a predetermined number of image parts from the memory unit based on a plurality of attribute values included in the derived metadata, and generates the image by combining the predetermined number of image parts.
[0083] [3] The image generating device described in [2], wherein each of the multiple image parts stored in the memory unit is given a name consisting of one or more character strings that represent the image part, and the generation unit selects a predetermined number of image parts that have character strings that match or are similar to the multiple attribute values included in the derived metadata, and generates the image by combining the predetermined number of image parts.
[0084] [4] An image generating device as described in [2] or [3], further comprising a modification unit that modifies at least a part of the image, wherein each of the plurality of image parts stored in the memory unit is composed of a plurality of sub-parts, each of the plurality of sub-parts is assigned an identifier that identifies each sub-part, and the modification unit changes the color of the sub-part corresponding to the identifier specified by a user.
[0085] [5] An image generating device according to any one of [1] to [4], wherein when the metadata is modified by a user, the generation unit generates an image based on the modified metadata.
[0086] [6] The image generating device according to any one of [1] to [5], wherein the metadata is metadata written in JSON (JavaScript Object Notation) format.
[0087] [7] The image generating device according to any one of [1] to [6], wherein the language model is a large-scale language model (LLM). [Explanation of symbols]
[0088] 1: user, 10: image generating device, 11: user device, 201: parts management unit, 202: naming unit, 203: ID (identifier) assignment unit, 204: text information acquisition unit, 205: prompt acquisition unit, 206: metadata derivation unit, 207: image generating unit, 208: output control unit, 210: data storage unit, 211: parts group, 212: language model, 213: image content group
Claims
1. an acquisition unit that acquires text information and a prompt, the prompt being an instruction to output metadata including a plurality of attribute values, the plurality of attribute values being attribute values for each of a plurality of attributes, so as to semantically match the text information; a derivation unit that derives metadata for the text information by inputting the text information and the prompt into a language model; a generator for generating an image based on the derived metadata; An image generating device having the following:
2. Further comprising a storage unit for storing a plurality of image parts, The image generating device according to claim 1 , wherein the generation unit selects a predetermined number of image parts from the storage unit based on a plurality of attribute values included in the derived metadata, and generates the image by combining the predetermined number of image parts.
3. Each of the plurality of image parts stored in the storage unit is assigned a name consisting of one or more character strings expressing the image part, The image generating device according to claim 2 , wherein the generation unit selects the predetermined number of image parts having character strings that match or are similar to the multiple attribute values included in the derived metadata, and generates the image by combining the predetermined number of image parts.
4. A modification unit that modifies at least a part of the image, Each of the plurality of image parts stored in the storage unit is composed of a plurality of subparts, and an identifier for identifying each of the plurality of subparts is assigned to each of the subparts; The image generating device according to claim 2 , wherein the change unit changes a color of a sub-part corresponding to an identifier designated by a user.
5. The image generating device according to claim 1 , wherein, when the metadata is modified by a user, the generating section generates an image based on the modified metadata.
6. The image generating device according to claim 1 , wherein the metadata is metadata written in a JavaScript Object Notation (JSON) format.
7. The image generating apparatus according to claim 1 , wherein the language model is a large-scale language model (LLM).
8. obtaining text information and a prompt, where the prompt is an instruction to output metadata including a plurality of attribute values, the attribute values being for each of a plurality of attributes, such that the metadata semantically matches the text information; deriving metadata for the text information by inputting the text information and the prompt into a language model; generating an image based on the derived metadata; An image generating method comprising:
9. On the computer, an acquisition step of acquiring text information and a prompt, the prompt being an instruction for outputting metadata including a plurality of attribute values, the plurality of attribute values being attribute values for each of a plurality of attributes, so as to semantically match the text information; a deriving step of deriving metadata for the text information by inputting the text information and the prompt into a language model; a generation step of generating an image based on the derived metadata; An image generation program that executes the following:
Citation Information
Patent Citations
Content management device, content management method, and program
JP2019204184A
JPP7216255B
Information processing device, information processing method, and information processing program
WO2023053364A1
Cited By
Text-to-image generation model evaluation method and system based on multi-modal large model
CN120071055A
Method and System for Evaluating Text-to-Image Generation Models Based on Multimodal Large Language Models
CN120071055B
Electronic device and video generation method using same
WO2026141985A1
