Image generation and search device, image generation and search system, and image generation and search method

Abstract

To provide an image generation and search device which enables a user to easily find a desired image from many image generation results.SOLUTION: Similarity is calculated from a sentence feature quantity and an image feature quantity and is used to search for a similar image similar to a search object from an image sentence database, and an image generation process held in the image sentence database is visualized.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image generation and search device, an image generation and search system, and an image generation and search method.

Background Art

[0002] By utilizing artificial intelligence (AI) technology, not only has it become possible to accurately recognize the objects shown in an image (see Non-Patent Document 1), but also techniques for generating the image itself have been rapidly developed.

[0003] The adversarial generative network (GAN) proposed in 2014 consists of two neural networks, a discriminative network and a generative network, and is a model that learns training images so that both compete with each other, and generates an image close to the real thing from the generative network.

[0004] At the beginning of 2021, research on a method for controlling the output of an image generation model by utilizing a large language model has progressed (see Non-Patent Document 2), and in 2022, high-quality image generation using a diffusion model, which is a method different from GAN, has become possible (see Non-Patent Document 3).

[0005] Recent image generation models can generate images of complex concepts in accordance with the content of a descriptive text input in natural language. Image generation by AI can be utilized not only for content creation but also for collecting training data for AI itself. In particular, since it is difficult to collect videos of rare cases such as disasters and accidents, if the generation model can be appropriately controlled to generate a large number of images suitable for the use case, the development of an image recognition system can be accelerated.

[0006] For example, the image generation device of Patent Document 1 is characterized by having an acquisition unit that acquires information about ingredients input by the user and randomly generated random number data, and a trained generation unit that uses a trained model for generating cooking images, taking the information about ingredients and the random number data acquired by the acquisition unit as input, to generate cooking images, thereby easily providing cooking images that allow for creativity and ingenuity, which can be helpful in creating a wide variety of dishes.

[0007] Furthermore, the information processing device described in Patent Document 2 easily generates image data according to the user's wishes by having a computer perform the following steps: display a plurality of first image data; accept the selection of a plurality of second image data from the plurality of first image data according to the user's operation; generate a plurality of third image data having characteristics corresponding to a combination of the characteristics of the plurality of second image data; and switch and display the plurality of third image data in predetermined units according to the user's operation. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2019-153223 (Patent No. 6865705) [Patent Document 2] Japanese Patent Publication No. 2020-102041 [Non-patent literature]

[0009] [Non-Patent Document 1] GLIPv2: Unifying Localization and VL Understanding [Non-Patent Document 2] Learning Transferable Visual Models From Natural Language Supervision [Non-Patent Document 3] High-Resolution Image Synthesis with Latent Diffusion Models [Overview of the project] [Problems that the invention aims to solve]

[0010] In the image generation device described in Patent Document 1, the user can control the content of the image output by the image generation model by inputting the names of ingredients in natural language. However, because the output image changes due to randomly generated random numbers, it takes time to check the results. Also, if the expected image is not generated, it is necessary to repeat the process many times or try different combinations of ingredient names through trial and error.

[0011] In the information processing device described in Patent Document 2, the user selects a generated image, obtains the generation vector that forms the basis of the generated image, and generates a new image using that generation vector, thereby producing an image intended by the user. However, since the user must judge the generated image, it takes time to make judgments when a large number of images are generated. Furthermore, since the feedback to the generation process is in a vector format that the user cannot decipher, it cannot be applied to image generation using natural language as input.

[0012] The object of the present invention is to provide an image generation and search device that allows a user to easily find a desired image from a large number of generated images. [Means for solving the problem]

[0013] An image generation and search apparatus according to one aspect of the present invention is an image generation and search apparatus having a processor, the image generation unit acquires a new generated image by image generation processing from an input sentence and an input image input by the processor, the sentence registration unit calculates a sentence feature amount from the input sentence by the processor, the image registration unit calculates an image feature amount from the input image and the generated image by the processor, an image-sentence database that holds the input sentence, the input image, the generated image, the image feature amount, the sentence feature amount, and the input-output relationship in the image generation processing as an image generation process, a search unit that calculates a similarity from the sentence feature amount and the image feature amount by the processor, and searches for a similar image similar to a search target from the image-sentence database using the similarity, and a display unit that visualizes the image generation process held in the image-sentence database by the processor.

Effect of the Invention

[0014] According to one aspect of the present invention, it is possible to provide an image generation and search apparatus that enables a user to easily discover a desired image from a large number of image generation results.

Brief Description of the Drawings

[0015] [Figure 1] It is a block diagram showing the configuration of an image generation / search system. [Figure 2] It is a block diagram showing the hardware configuration of an image generation / search system. [Figure 3] It is a diagram showing the structure of an image / sentence database. [Figure 4A] It is a diagram for explaining image generation processing. [Figure 4B] It is a diagram for explaining image generation processing. [Figure 4C] It is a diagram for explaining image generation processing. [Figure 5] It is a flowchart showing image generation and database construction processing. [Figure 6A] It is a diagram for explaining image search processing. [Figure 6B]This is a diagram for explaining image search processing. [Figure 7] This is a flowchart showing image search processing. [Figure 8] This is a diagram for explaining visualization of the image generation process. [Figure 9] This is a flowchart showing the visualization processing of the image generation process. [Figure 10] This is a diagram for explaining filtering of the image generation result by similar image search. [Figure 11] This is a flowchart showing the filtering processing of the image generation result by similar image search. [Figure 12] This is a diagram for explaining filtering of the image generation result by image recognition. [Figure 13] This is a flowchart showing the filtering processing of the image generation result by image recognition. [Figure 14] This is a diagram for explaining assistance for text input by similar image search. [Figure 15] This is a flowchart showing the assistance processing for text input by similar image search. [Figure 16] This is a diagram for explaining assistance for mask image input by similar image search. [Figure 17] This is a flowchart showing the assistance processing for mask image input by similar image search. [Figure 18] This is a diagram showing an example screen of the image generation / search system. [Figure 19] This is a sequence diagram showing the processing of the entire image generation / search system.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. This embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention. The same reference numerals are assigned to common configurations in each figure.

Examples

[0017] The image generation and search device 104 of this embodiment 1 automatically generates images that match the content of a text based on the input text or images. It also extracts feature quantities from the input or generated text and images and constructs an image and text database 110 for similar image search by combining this with information on the input and output relationships of the image generation. By using the database search function, the user can check the process of past image generation and efficiently find the desired image from a large number of generated images.

[0018] Figure 1 is a block diagram showing an example configuration of the image generation and search system 100 of Example 1.

[0019] Possible use cases for the image generation and search system 100 include content creation and data collection for machine learning, but are not limited to these. The following describes each component.

[0020] The image generation and search system 100 automatically generates images that correspond to the content of text from text or images entered by the user and registers them in the image and text database 110. The image generation and search system 100 consists of an image storage device 101, an input device 102, a display device 103, and an image generation and search device 104. The image storage device 101 is a storage medium that stores still image and video data or associated bibliographic information, and is configured using a hard disk drive built into a computer, or a storage system connected via a network such as a NAS (Network Attached Storage) or SAN (Storage Area Network). The image storage device 101 may also be a cache memory that temporarily holds data continuously input from the imaging device.

[0021] The input device 102 is an input interface, such as a mouse, keyboard, or touch device, for transmitting user operations to the image generation / search device 104. It may also be a voice input interface that allows text input via voice recognition. The display device 103 is an output interface, such as a liquid crystal display, and is used for displaying search results from the image generation / search device 104, interactive operations with the user, etc.

[0022] The image generation and search device 104 is a device that performs image generation processing to generate images that match the content of text from text or images input by the user, registration processing to create a database of input and generated images and text, and search and display processing to process the generation results and present them to the user.

[0023] The image generation and search device 104 includes a text input unit 105, an image input unit 106, an image generation unit 107, a text registration unit 108, an image registration unit 109, an image / text database 110, a search unit 111, a display unit 112, and an image generation assistance unit 113.

[0024] The following describes the image generation and database registration processes. Further details of the process are also explained in the flowchart in Figure 5.

[0025] In the image generation process, images are automatically generated from text entered by the user via the input device 102, based on the content of the text. Alternatively, existing images stored in the image storage device 101 can be used as input and modified to match the text content. In the database registration process, feature quantities for similar image search are extracted from the input text, images, and generated images, and registered in the image / text database 110.

[0026] Features are numerical data in multidimensional vector form, and the similarity of the source data from which features are extracted can be determined by calculating the cosine similarity between two vectors. By using a model trained on image-text pairs for feature extraction, similarity can be calculated between images and images, images and text, and text and text using the same procedure. In addition, the database registration process registers the input / output relationship of the image generation process in the image / text database 110. This makes it possible to present the process of past image generation processes.

[0027] The text input unit 105 receives text entered by the user using the input device 102 and converts it into a data format used internally by the image generation and search device 104. The image generation method targeted by this invention can basically take natural language text as input, but there are often constraints on the type of language and the length of the text.

[0028] Furthermore, it may accept a predetermined set of words rather than natural language. In addition, it may emphasize or weaken words according to specific description rules. Therefore, the text input unit 105 transforms the input text according to the image generation model used by the image generation unit 107. For example, a Japanese sentence is translated into an English sentence. If the image generation unit 107 uses multiple machine learning models, it outputs multiple transformed sentences that are converted according to the constraints of each model.

[0029] The image input unit 106 receives still image data or video data from the image storage device 101 and converts it into a data format used internally by the image generation and retrieval device 104. For example, if the data received by the image input unit 106 is video data, the image input unit 106 performs video decoding to decompose it into frames (still image data format). If the data is stroke information entered using a mouse or touch device, it draws these strokes onto the image.

[0030] The image generation unit 107 automatically generates images that correspond to the content of one or more sentences using an image generation model. The image generation model uses an image generation algorithm like that described in Non-Patent Document 3, and uses model parameters obtained by training a large number of images and their descriptions using deep learning. Therefore, the generated images vary greatly not only depending on the input sentences but also on the image generation algorithm and training data. Furthermore, in algorithms that use random noise as the initial value, the generated images also change depending on the random seed.

[0031] Depending on the image generation algorithm, it may be possible to input an image and modify it to match a sentence, or to input a mask image to modify only a part of an image. Furthermore, it may be possible to input two or more sentences and generate an image representing an intermediate concept, or to input a negative sentence and generate an image that does not contain elements of that sentence. In this way, the image generation unit 107 can generate a wide variety of images depending on the input. Additionally, by using multiple image generation models, it is possible to generate a large number of different images at once from user-inputted conditions.

[0032] The text registration unit 108 extracts features from the input text acquired by the text input unit 105 and used for image generation. These features are typically given as fixed-length vector data, and the similarity between the two original data can be determined by calculating the Euclidean distance or cosine similarity between the two vectors. Furthermore, by using the language-image feature extraction model described in Non-Patent Literature 2, comparable features can be obtained from text and images.

[0033] The image registration unit 109 extracts features from the input image acquired by the image input unit 106 or from the generated image generated by the image generation unit 107. The image registration unit 109 of the present invention extracts features from the image using a feature extraction model similar to that of the text registration unit 108.

[0034] The image / text database 110 stores input texts and images or generated images, their feature quantities, and information on the input / output relationships of the image generation process. The image / text database 110 can search for registered data that satisfies given conditions or read data with a specified ID in response to queries from each part of the image generation / search device 104. It can also output registered data similar to the query by using feature quantities extracted from images and texts. The details of the structure of the image / text database 110 will be described later in Figure 3.

[0035] The above describes the operation of each part in the image generation and database registration processes of the image generation and search device 104. Next, the operation of each part in the search and display processes of the image generation and search device 104 will be explained. The details of the process will also be explained in the flowchart in Figure 7.

[0036] In the search and display process, the system uses the search criteria specified by the user via the input device 102 to search the image and text database for images that match the search criteria and displays the information on the display device 103. For example, the system can search the database using a search query given in text or image format, and display the registered images in the database sorted by similarity.

[0037] The text used to generate the image can be used as the search query, or the resulting image can be used as the query. This allows the user to obtain desired images not only from the image generated in the most recent image generation procedure, but also from images generated in the past. In the search and display process, the input and output relationships of image generation registered in the image and text database 110 can also be visualized. This allows the user to generate new images by referring to past generation processes.

[0038] The search unit 111 retrieves image and text data from the image and text database 110 using the specified search conditions. If the query is given as a conditional expression, it returns registered data that matches the conditions. If the query is given as vector data, it calculates the similarity between the vectors and returns the data sorted according to the similarity. In addition, the search unit 111 can process the images and text retrieved from the image and text database 110 by performing necessary analysis processes to obtain the information the user needs.

[0039] The display unit 112 displays data acquired from the image / text database 110 on the display device 103. For example, it may display the generated image and similar images side by side, or it may display a graph showing the input / output relationship of the image generation process.

[0040] The image generation assistance unit 113 uses the results from the search unit 111 to assist the user in generating images. For example, it can select a single image specified by the user from similar images obtained by the search unit 111 as the initial image for generating a new image, or it can reuse previously used text as input text. Also, for example, when using multiple image generation models, it may activate only the image generation model that has produced the most results that the user deems desirable and generate a new image from that model.

[0041] The above describes the operation of each part in the search and display processing of the image generation and search device 104. Note that the image generation and registration processing and the search and display processing of the image generation and search device 104 are processes that are repeatedly performed according to user instructions, and the contents of the image and text database 110 are added and updated sequentially, so that the processing content of each part that uses the registered data changes accordingly. Furthermore, if mutual exclusion control for database updates is properly implemented, multiple users can access and use it simultaneously.

[0042] Figure 2 is a block diagram showing an example of the hardware configuration of the image generation and search system 100 in this embodiment. The image generation and search device 104 comprises an interconnected processor 201 and a storage device 202. The storage device 202 is composed of any type of storage medium. The storage device 202 is composed of a combination of storage devices such as semiconductor memory and hard disk drives.

[0043] The functional units shown in Figure 1, such as the text input unit 105, image input unit 106, image generation unit 107, and text registration unit 108, are realized by the processor 201 executing the processing program 203 stored in the storage device 202. In other words, the processing performed by each functional unit is executed by the processor 201 based on the processing program 203.

[0044] Furthermore, the data in the image and text database 110 is stored in the storage device 202. Note that if the image generation and search system 100 is configured with multiple devices for purposes such as load balancing, the device containing the image and text database 110 and the device executing the processing program 203 may be physically different devices connected by a network, or the processing program 203 may be executed simultaneously on multiple devices as long as the integrity of the data recorded in the image and text database 110 is maintained.

[0045] The image generation and retrieval device 104 further includes a network interface device (NIC) 204 connected to the processor 201. The image storage device 101 is assumed to be a NAS or SAN connected to the image generation and retrieval device 104 via the network interface device 204. The image storage device 101 may also be included in the storage device 202.

[0046] Figure 3 is an explanatory diagram showing the configuration and data example of the image / text database 110 in this embodiment. Note that in this embodiment, the information used by the system is not dependent on the data structure and may be represented in any data structure. While Figure 3 shows an example in table format, information can be stored in a data structure appropriately selected from, for example, tables, lists, databases, or queues.

[0047] The image / text database 110 includes, for example, an image / text information table 300. The table configuration and field configuration of each table in Figure 3 are examples, and tables and fields may be added depending on the application, for example. Also, the table configuration may be changed as long as similar information is held. For example, the image / text information table 300 may be divided into an image information table and a text information table, or information on input / output relationships for image generation may be managed in a separate table.

[0048] The image / text information table 300 consists of an ID field 301, a type field 302, an image field 303, a text field 304, an input text ID field 305, an input image ID field 306, an input mask ID field 307, and a feature field 308.

[0049] The ID field 301 holds the identification number of the image / text information. The Type field 302 holds the type of information. The types of information can be, for example, text, image, mask image, etc. Here, a mask image is an image in which information about the region that will change the input image during image generation is recorded. The Image field 303 holds the binary data of the image if the type is an image. This may be a file path on file storage, as long as the binary data of the image is accessible.

[0050] The text field 304 holds the text string if the type is text. The input text ID field 305 holds the ID managed in the image / text information table 300 for the input text used for image generation. The input image ID field 306 holds the ID managed in the image / text information table 300 for the input image used for image generation. The input mask ID field 307 holds the ID managed in the image / text information table 300 for the input mask image used for image generation. The feature field 308 holds a numerical vector representing the features extracted from the image or text.

[0051] The image / text information table may have additional fields added as needed. For example, fields such as the time the data was registered, information about the model used for image generation, various parameters used for image generation (e.g., random number seed), and user-generated image ratings may be added. Also, if an image generation model that accepts multiple input texts or images is used, the input text ID field 305, input image ID field 306, and input mask ID field 307 may be made to hold multiple IDs. Multiple feature fields 308 may also be provided to store the results calculated using multiple feature extraction methods, allowing the user to select which features to use during a search.

[0052] Figures 4A, 4B, and 4C illustrate the overview of the image generation process of the image generation unit 107 in this embodiment. The image generation process is a process that automatically generates images according to the content of the input text, and known algorithms such as those shown in Non-Patent Literature 3 can be used. The image generation process of the image generation unit 107 in this embodiment uses three main types of input / output patterns.

[0053] In text-to-image generation, one or more images 402 that correspond to the content of a single input text 401 are generated (see Figure 4A).

[0054] In image generation from text and images, the input image 403 is modified according to the content of the text (see Figure 4B).

[0055] In image generation from text, images, and masks, the input image is modified in the area specified by the mask image 404 (see Figure 4C). In Figure 4C, the black-filled area of ​​the mask image 404 retains the information of the input image, while the white-filled area generates an image according to the input text. The mask image can be given as a binary value indicating whether or not to apply modification, or as a continuous value representing the intensity of the modification.

[0056] Although three types of patterns have been described above, the image generation process of the image generation unit 107 in this embodiment is not limited to these. By adding fields to the image / text information table 300, derived image generation algorithms with different numbers of inputs and outputs may be used. For example, by inputting multiple sentences, it may be possible to generate an image representing an intermediate concept between them. Furthermore, by resizing or cropping the input image and inputting it into the image generation model, it may be possible to support outpainting, which involves drawing outside the frame of the input image.

[0057] Figure 5 is a diagram illustrating the image generation and database registration process flow. In this embodiment, the image generation and search system 100 performs image generation when the user inputs text for image generation, and automatically performs database registration unless otherwise specified. The following describes each step in Figure 5.

[0058] The text input unit 105 receives text-like information from the input device 102 and converts it into text data usable within the system as needed (S501). The text registration unit 108 calculates feature quantities from the text data acquired in step S501 and registers them in the image / text database 110 (S502).

[0059] The image generation and retrieval device 104 executes step S504 if it uses an initial image as input for the image generation process, and otherwise executes step S506 (S503). The image input unit 106 acquires an initial image to be used as input for the image generation process from the image storage device 101 and converts it into image data usable within the system as needed (S504).

[0060] The image registration unit 109 calculates image features from the initial image data and registers them in the image / text database 110 (S505). The image generation assistance unit 113 receives image generation conditions from the input device 102 (S506). The image generation conditions are parameters such as the input pattern for the image generation process described in Figure 4, the type of generation model, the random number seed, and the number of images to output. If a mask image is used in the generation process, the unit receives information about the mask area from the input device 102 and generates the mask image.

[0061] The image generation and search device 104 executes step S508 if a mask image is used in the image generation process, and otherwise executes step S510 (S507).

[0062] The image registration unit 109 calculates feature quantities from the mask image generated in step S506 and registers them in the image / text database 110 (S508). The image generation unit 107 generates an image from the text input in step S501 according to the conditions input in step S506 (S509). Depending on the specified conditions, the initial image input in step S504 and the mask image generated in step S506 are added to the input of the image generation process.

[0063] The image registration unit 109 calculates feature quantities from the generated image obtained in step S509 and registers them in the image / text database 110 (S510). In addition, when registering the generated image, information about the input data used in the generation process is also recorded.

[0064] Figures 6A and 6B illustrate the overview of the similar image search process of the search unit 111 in this embodiment. The image generation and search device 104 in this embodiment extracts mutually comparable feature quantities from images and text and stores them in the image / text database 110. Therefore, by comparing feature quantities, similar data can be obtained from the image / text database 110 using an image or text as a query.

[0065] The search process allows for (a) image search from query text, (b) image search from query image, (c) text search from query text, and (d) text search from query image. Figure 6A shows an example of (a) for obtaining similar images. Figure 6B shows an example of (b) for obtaining similar images.

[0066] As shown in Figure 6A, in image search from query text, features are extracted from the input query text and used as query features. The similarity between registered features and query features is calculated for data of type "image" from the registered data in the image / text database 110. Feature data are multidimensional vector data, and the similarity between two vectors can be calculated, for example, using cosine similarity.

[0067] Result 601 is given by the ID managed in the image / text information table 300 and the similarity obtained by vector calculation. When performing an image search from a query text, the emphasis is on whether or not the concept of the text is included, so the image characteristics of the search results tend to differ significantly.

[0068] As shown in Figure 6B, the procedure is similar for image searches using a query image: query features are calculated from the input query image, and the image / text database 110 is searched. When using a query image, the features include various image characteristics that cannot be expressed in text, so there is a strong tendency for images with similar atmosphere and composition to appear in the search results. On the other hand, there is a tendency for images that do not contain the intended concept to appear in the search results.

[0069] Figure 7 is a diagram showing the processing flow of the similar image search in the search unit 111 of this embodiment. The steps in Figure 7 will be described below.

[0070] The search unit 111 executes step S702 if the search query is text, step S703 if the search query is image, and step S704 if the search query is specified by a database ID (S701).

[0071] The search unit 111 executes step S706 if there are search refinement conditions, and otherwise executes step S707 (S705). The search unit 111 extracts features from the query text (S702). The feature extraction process is performed using the same method as the text registration unit 108.

[0072] The search unit 111 extracts features from the query image (S703). The feature extraction process is performed using the same method as the image registration unit 109. The search unit 111 obtains the features of the specified ID from the image / text database 110 (S704).

[0073] The search unit 111 narrows down the records registered in the image / text database 110 to those to be used for similarity calculation using the specified filtering conditions (S706). For example, if the filtering condition is set to "images only", the search results output after similarity calculation will consist only of images.

[0074] The search unit 111 calculates the similarity of all target records for similarity calculation with the feature obtained in step S702, step S703, or step S704 (S707). The cosine similarity of two vectors can be used for similarity calculation. Alternatively, by pre-storing data with high similarity, a known approximate nearest neighbor search method can be applied to quickly search only for records with high similarity without having to calculate the similarity of all records.

[0075] The search unit 111 sorts the search results using similarity and outputs a specified number of results (S708). In similar image search, it is common to output search results in descending order of similarity, but depending on the purpose, it may be possible to output them in descending order of similarity, or to set upper or lower limits on the similarity. Alternatively, the search results may be thinned out so that the similarity between them falls below a certain level in order to output more diverse search results.

[0076] Figure 8 is a diagram illustrating the overview of the visualization of the image generation process in the display unit 112 of this embodiment. The image / text database 110 of this embodiment stores information on the input / output relationships of the image generation process, and by visualizing this information, the user can learn effective procedures from past image generation.

[0077] Figure 8 visualizes the generation process using a directed graph consisting of nodes and edges. Each node represents a record registered in the image / text database 110, displaying either an image thumbnail or text depending on the type. Edges represent input / output relationships, with the node connected to the starting point of the arrow representing the input data and the node connected to the ending point representing the generated image. For example, the process by which image 802 is generated from text 801 is represented by edge 803. Directed graphs allow for a visual understanding of any image generation process.

[0078] For example, it can be seen that text 801 is used in numerous image generation processes, and that image 806 was generated using text 803 and mask 805 on image 804, which was obtained through several image generation processes. In graph visualization, visibility may deteriorate as the amount of data increases, but generally accepted simplified display methods can be applied, such as displaying only the area around the node of interest or grouping together generation processes under the same conditions.

[0079] Furthermore, since each node represents a record with features that can be used to calculate similarity, it is possible to control the placement of nodes with similarity to each other. For example, since sentences 801 and 803 have high similarity, the graph's rendering can be controlled by assigning a virtual edge 808 to them.

[0080] Figure 9 is a diagram showing the processing flow of the visualization of the image generation process of the display unit 112 in this embodiment. The steps in Figure 9 will be described below.

[0081] The search unit 111 retrieves information about the record to be drawn, as specified by the user, from the image / text database 110 (S901).

[0082] The display unit 112 adds a node to the graph using the information of the record to be drawn acquired in step S901 (S902).

[0083] The display unit 112 executes steps S904 to S906 (S903) with respect to the input records in the generation process held by the drawing target record acquired in step S901. Here, the input records are the ID records held in the input document ID field 305, input image ID field 306, and input mask ID field 307 of the image / text information table 300.

[0084] The display unit 112 executes step S906 if the input record has already been plotted on the graph, otherwise it executes step S905 (S904).

[0085] The image generation and search device 104 performs the processing flow shown in Figure 9, using the input record as the record to be drawn (step S905).

[0086] The display unit 112 adds an edge between the node of the record to be drawn and the node of the input record (S906). The edge is a directed edge that goes from the node of the input record to the node of the record to be drawn.

[0087] The display unit 112 executes step S908 (S907) after processing all input records. The display unit 112 executes step S909 if the type of record to be displayed is a document and there are nodes of document records with a high degree of similarity; otherwise, it executes step S 910 Execute (S908).

[0088] The display unit 112 adds a virtual edge between the node of the record to be drawn and the node of the document record with a high degree of similarity (S909). The virtual edge is added to control the drawing position of the node and may or may not be displayed on the screen presented to the user.

[0089] Display section 112 This optimizes the node placement according to the edge connectivity (S910). Known graph plotting techniques can be used for optimization. For example, a method can be used to iteratively find the arrangement where the basic repulsive forces between each node balance the attractive forces between nodes connected by edges. [Examples]

[0090] The image generation and search device 104 of Example 1 made it possible for users to efficiently find desired images by searching for generated images stored in an image and text database using text or images, and by visualizing the image generation process. However, when generating a new image, it was necessary for the user to explicitly perform an image search or check the visualization graph of the generation process.

[0091] The image generation and search device 104 in Example 2 automatically analyzes a large number of images generated under predetermined conditions, thereby prioritizing the presentation of images that the user expects. This allows for efficient image acquisition even when generating images under new conditions.

[0092] Figure 10 is a diagram illustrating the filtering process of image generation results based on similarity in Example 2. The image generation unit 107 can generate a large number of images in response to a single image generation request from the user by changing the image generation model and image generation parameters.

[0093] The image generation and search device 104 of Example 2 calculates similarity to the generated result 1001 using a pre-set query set 1002, calculates a score, and outputs images with a score above a predetermined threshold as the filtered result 1003.

[0094] The query set 1002 can be provided as images or text, and users can add them interactively. The score of the images to be evaluated is calculated, for example, by calculating the similarity between each query and the image to be evaluated, and adding them together with weights as needed. By setting negative values ​​for the query weights, images with low similarity to the query can be scored higher. For example, by collecting generated images with low similarity to the query, a wide variety of images can be obtained.

[0095] Figure 11 is a diagram illustrating the processing flow for filtering the image generation results by similarity in Example 2. The following describes each step in Figure 11.

[0096] The image generation unit 107 generates a predetermined number of images using a predetermined combination of image generation models and image generation parameters (S1101).

[0097] The search unit 111 obtains filtering conditions from the input device 102 (S1102). The filtering conditions include queries consisting of one or more images or texts, the weight of each query, a score threshold, and so on.

[0098] The search unit 111 executes steps S1104 to S1105 for each query specified by the filtering conditions obtained in step S1102 (S1103). The search unit 111 performs a similar image search process using the query image or query text on the generated image set generated in step S1101 and obtains the search results (S1104).

[0099] The search unit 111 adds the similarity to the query obtained in step S1104 to the score of each generated image (S1105). If the query was assigned a weight in the filtering conditions obtained in step S1102, the weight is applied before the addition.

[0100] The search unit 111 executes step S1107 after processing all queries (S1106). The search unit 111 sorts the generated images in descending order of total score and outputs a predetermined number of them (S1107). If a score threshold is set in the filtering conditions obtained in step S1102, it outputs generated images that are equal to or above the threshold.

[0101] Figure 12 is a diagram illustrating the filtering process of the image generation results by image recognition in Example 2. The image generation unit 107 can automatically generate images that conform to the content of the input text, but depending on the image generation model and image generation parameters, the generated images may not necessarily include all the concepts contained in the input text.

[0102] The image generation and search device 104 in Example 2 automatically verifies, through image recognition processing, whether each concept contained in the input text is also included in the generated image. In the example in Figure 12, objects are detected in the generated image using image recognition processing, and a score is calculated based on whether the objects "cat," "apple," and "table" contained in the input text are included, and filtering is performed accordingly.

[0103] Figure 13 is a diagram illustrating the processing flow for filtering the image generation results by image recognition in Example 2. The following describes each step in Figure 13.

[0104] The image generation unit 107 generates a predetermined number of images using the input text and a predetermined combination of image generation models and image generation parameters (S1301).

[0105] The search unit 111 obtains filtering conditions from the input device 102 (S1302). The filtering conditions are the object detection algorithm used for image recognition and its parameters. A known algorithm shown in Non-Patent Literature 1 can be used as the object detection algorithm.

[0106] The search unit 111 extracts a list of words for objects that can be detected by the object detection algorithm specified in step S1302 from the input text used for image generation in step S1301 (S1303).

[0107] The search unit 111 executes steps S1305 to S1306 for each query specified by the filtering conditions obtained in step S1302 (S1304).

[0108] The search unit 111 performs object detection processing on each generated image generated in step S1301 (S1305). The search unit 111 calculates a score from the degree of match between the list of objects detected in step S1304 and the word list obtained in step S1303, and adds it to the total score of each generated image (S1306).

[0109] The search unit 111 executes step S1308 after processing all queries (S1307). The search unit 111 sorts the generated images in descending order of total score and outputs a predetermined number of them (S1308). If a score threshold is set in the filtering conditions obtained in step S1302, it outputs generated images that are equal to or above the threshold.

[0110] Figures 11 and 12 show an example where the degree of agreement between the input text and the generated image content is determined using an object detection algorithm. However, the image generation and search device 104 of Example 2 can use any image recognition algorithm as long as the degree of agreement between the input text and the generated image content can be determined.

[0111] For example, the degree of similarity can be determined by comparing the text generated using an image description generation algorithm with the input text. Alternatively, even when using an object detection algorithm, the degree of similarity can be calculated by considering the positional relationship of the objects. For instance, based on the relationship between "cat" and "on the table," the score of generated images where "cat" is positioned on top of "table" could be increased in the object detection results. [Examples]

[0112] The image generation and search device 104 of Example 2 can efficiently present images desired by the user from a large number of images generated under predetermined conditions through image analysis processing. On the other hand, the image generation conditions need to be adjusted by the user through trial and error.

[0113] The image generation and search device 104 of Example 3 can present the user with candidate text and mask images to be input for image generation by searching past generated images stored in the image and document database 110.

[0114] Figure 14 illustrates the text input assistance using image search in Example 3. Using the input text, the generated image is used as a query to search the image / text database 110 for similar images generated in the past. Since the image / text database 110 stores information about the input and output processes of the image generation process, the input text used in the image generation process can be obtained from each image in the search results.

[0115] In the image generation assistance unit 113 of Example 3, keywords used for image generation can be extracted by analyzing text obtained from similar images. Keyword extraction can simply involve listing the words that appear, or it can involve calculating a word score using an arbitrary statistical indicator and changing the display order according to the score. For example, word frequency (TF) can be used as an indicator, or the statistical quantity TF-IDF, which is the product of TF and inverse document frequency (IDF), can be used as an indicator. Here, TF is the frequency of words included in the similar image search results, and IDF is the reciprocal of the number of documents containing the word in question from all records in the image / text database 110. Using TF-IDF makes it possible to extract words that appear more frequently than usual in the similar image search results.

[0116] Figure 15 is a diagram illustrating the processing flow of the image search-based text input assistance in Example 3. The following describes each step in Figure 15.

[0117] The image generation unit 107 generates an image from the input text (S1501). Step S1501 is equivalent to the image generation process described in Figure 5. The image generation assistance unit 113 terminates processing if it determines that the user has obtained the desired image; otherwise, it executes step S1503 (S1502).

[0118] The image search unit 111 searches the image / text database 110 for similar images using the generated image obtained in step S1501 as a query (S1503). Step S1503 is equivalent to the image search process described in Figure 7.

[0119] The image generation assistance unit 113 executes step S1505 for each similar image acquired in step S1503 (S1504).

[0120] The image search unit 111 retrieves related text for similar images from the image / text database 110 (S1505). The related text is, for example, the data stored in the text field 304 of the record that corresponds to the ID stored in the input text ID field 305 of the record for the similar image.

[0121] The image generation assistance unit 113 executes step S1507 after processing all similar images (S1506). The image generation assistance unit 113 extracts a set of keywords from the related text obtained in step S1505 (S1507). As described in the explanation of Figure 14, the keyword extraction method may be a simple enumeration of words, or it may be the result of scoring using statistical indicators such as TF or TF-IDF.

[0122] The image generation assistance unit 113 adds the keyword selected by the user via the input device 102 from the keyword set extracted in step S1507 to the input text (S1508). At this time, the keyword may simply be added to the end of the text, or a text obtained using a known text generation algorithm that generates text from keywords may be added.

[0123] Figure 16 is a diagram illustrating the mask input assistance by image search in Example 3. As explained in Figure 4C, the image generation and search device 104 of the present invention can generate an image with a part of the input image modified by inputting a mask image.

[0124] In Example 3, the image generation assistance unit 113, when generating an image from a revised text by adding new elements while leaving existing elements in the initial text intact, searches for similar images in the image / text database 110 using the revised text and calculates a frequency map of which regions in the image are most likely to contain the added elements. From this frequency map and the regions of existing elements in the image generated from the initial text, it determines candidate regions for placing the added elements. It also calculates the relative size of the added elements to the existing elements, selects one or more regions within the candidate regions where the relative size can be placed, and generates candidate mask images.

[0125] Figure 17 is a diagram illustrating the processing flow of mask input assistance using image search in Example 3. The following describes each step in Figure 17.

[0126] The image generation unit 107 generates an image from the initial text (S1701). Step S1701 is equivalent to the image generation process described in Figure 5. The image generation assistance unit 113 analyzes the modified text entered by the user and extracts existing elements that were also included in the initial text, as well as newly added elements (S1702).

[0127] The image generation assistance unit 113 detects the retained element objects extracted in step S1702 from the generated image acquired in step S1701 (S1703). The image generation assistance unit 113 reflects the object regions detected in step S1703 as retained regions in the mask image (S1704). Here, retained regions are regions that are not modified by the image generation process, and are represented as black-filled regions in the mask image of Figure 16.

[0128] The search unit 111 searches the image / text database 110 for similar images using the modified text obtained in step S1702 as a query (S1705). The image generation assistance unit 113 detects existing and additional object elements extracted in step S1702 from each similar image obtained in step S1705 (S1706).

[0129] The image generation assistance unit 113 aggregates the occurrence areas of the additional elements detected in step S1706 to create an occurrence frequency map, and updates the mask image so that the areas with high occurrence frequency become the correction areas (S1707). Here, the correction areas are the areas that are corrected by the image generation process, and are represented as areas filled in white in the mask image of Figure 16.

[0130] The image generation assistance unit 113 calculates the relative size of the additional element to the existing element from the object regions of the existing element and the additional element detected in step S1706 (S1708).

[0131] The image generation assistance unit 113 selects one or more regions from the correction region of the mask image obtained in step S1707 that contain the relative size of the additional elements calculated in step S1708, and generates a mask image with those regions as the correction region (S1709).

[0132] The image generation and search device 104 described in Examples 1 to 3 above can generate a large number of images that correspond to the content of the input text, and efficiently find the image desired by the user from the generated images. Furthermore, in the process of generating new images, efficient generation can be achieved by filtering and displaying generated images through image analysis processing, or by presenting candidate texts and mask images to be used for image generation.

[0133] Figure 18 shows an example of the configuration of the operation screen for image generation and image retrieval in the image generation and retrieval device 104 of Examples 1 to 3.

[0134] The image generation and search device 104 displays the processing results on the display device 103. The user communicates operation information to the image generation and search device 104 using the mouse cursor 1801 and other elements displayed on the screen via the input device 102. The screen consists of a text input field 1802, an image generation button 1803, a generated image display field 1804, a generation process visualization field 1805, a search condition field 1806, an image search button 1807, an image search results field 1808, a text candidate field 1809, a text addition button 1810, a mask candidate field 1811, and a mask setting button 1812. The example screen configuration is just one example, and the screen can be configured by freely arranging these elements.

[0135] Figure 19 is a sequence diagram showing the process of image generation and image retrieval in the image generation and retrieval device 104 of Examples 1 to 3.

[0136] Figure 19 specifically shows the processing sequence between the user 1900, the image storage device 101, the computer 1920, and the image / text database 110 in each process of the image generation / search system 100 described above. The sequence in Figure 19 is broadly divided into the input assistance processing described in Example 3, the image generation / database registration processing described in Example 1, the output control processing, and the output control processing described in Example 2, and these sequences are repeatedly executed according to the user's request. The computer 1920 is the computer that implements the image generation / search device 104. The steps in Figure 19 will now be explained.

[0137] When user 1900 issues an image generation request (S1901), a series of processes related to image generation are initiated in computer 1920. If computer 1920 uses an input image for image generation, it requests the input image from image storage device 101 (S1902), and image storage device 101 returns the input image (S1903).

[0138] Computer 1920 sends a request to the image / document database 110 for a similar image search using the input image as a query (S1904), and the image / document database 110 returns the search results (S1905). Computer 1920 estimates additional sentence candidates from the sentences obtained from the search results (S1906) and presents the additional sentence candidates to user 1900 (S1907). The user modifies the input sentence using the presented candidates and inputs it into computer 1920 (S1908). Computer 1920 estimates mask candidates from the modified sentence and image search results (S1909) and presents the mask candidates to user 1900 (S1910). User 1900 inputs the mask image into computer 1920, referring to the presented mask image (S1911).

[0139] calculator 1920 This generates an image using the input text, image, and mask image (S1912). Computer 1920The computer extracts features from the text, images, mask images, and generated images used for generation (S1913) and registers them in the image / text database 110 (S1914). The image / text database 110 returns the ID of the record that has been registered (S1915). The computer 1920 uses the ID to retrieve the record information from the image / text database 110 (S1917), performs filtering and visualization processing (S1918), and presents the results to the user (S1919).

[0140] According to the above embodiment, it is possible to present information that makes it easy to find images. Furthermore, appropriate auxiliary information can be presented to the user when regenerating images. The effects of the above embodiment are expected to become more pronounced with the emergence of methods that can generate a large number of images with diverse concepts and variations using natural language.

[0141] More specifically, according to the above embodiment, it is possible to narrow down and present images that match the user's intent from a large number of images generated by text input. Furthermore, it is possible to assist in image generation input by searching the generated images stored in the image / text database and extracting conditions such as text candidates to be input for new image generation and presenting them to the user. In addition, the input and output relationships of image generation stored in the database can be visualized and presented to the user.

[0142] The embodiments relating to the present invention have been described above. It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are detailed explanations provided to make the present invention easier to understand, and are not necessarily limited to those having all the described configurations. Furthermore, it is possible to replace parts of the configuration of one embodiment with those of another embodiment, and to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with those of other embodiments.

[0143] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above configurations and functions may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0144] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and do not necessarily represent all control lines and information lines in the actual product. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0145] 100 Image Generation and Search Systems 101 Image storage device 102 Input device 103 Display device 104 Image generation and search device 105 Text Input Section 106 Image Input Section 107 Image generation unit 108 Document Registration Section 109 Image Registration Section 110 Image and Text Database 111 Search Section 112 Display section 113 Image generation support unit

Claims

1. An image generation and search device having a processor, The aforementioned processor includes an image generation unit that obtains a new generated image from the input text and input image through image generation processing, The processor includes a text registration unit that calculates text features from the input text, The processor includes an image registration unit that calculates image features from the input image and the generated image, An image-text database that stores the input text, the input image, the generated image, the image features, the text features, and the input / output relationships in the image generation process as part of the image generation process, The processor calculates similarity from the text features and image features, and the search unit uses the similarity to search for similar images from the image-text database that are similar to the search target. The processor includes a display unit that visualizes the image generation process stored in the image document database, An image generation and search device characterized by having the following features.

2. The aforementioned processor includes a text input unit that receives the input text, The processor includes an image input unit that receives the input image, The processor includes an image generation assistance unit that assists in the image generation process of the generated image, The image generation and search device according to claim 1, further comprising the above.

3. The search unit is controlled by the processor, The image text database is searched using the search query provided in the input text or the input image. The image generation and search device according to claim 1, characterized in that it sorts the searched similar images according to the degree of similarity.

4. The display unit is controlled by the processor, The image generation search device according to claim 1, characterized in that the generated image and the similar image are placed side by side to visualize the image generation process as a graph representing the input / output relationship.

5. The image generation unit is controlled by the processor, A predetermined number of the generated images are obtained as a set of generated images, The search unit is controlled by the processor, The image generation and search device according to claim 1, characterized in that, by filtering using the aforementioned similarity, it searches for similar images using a query image or query text on the generated image set and outputs the search results as a filtering result.

6. The search unit is controlled by the processor, In the filtering process, the similarity is calculated using a pre-set query set for the generated image to obtain a score. The image generation and search device according to claim 5, characterized in that it outputs the filtering result based on the score.

7. The image generation unit is controlled by the processor, A predetermined number of generated images are obtained from the input text, The search unit is controlled by the processor, The image generation and search device according to claim 1, characterized in that it recognizes whether the concept of a word contained in the input text is included in the generated image by filtering using image recognition, and outputs the recognition result as the filtering result.

8. The search unit is controlled by the processor, Object detection processing is performed on the generated image. A score is calculated from the degree of agreement between the object detection result in the object detection process and the word contained in the input text. The image generation and search device according to claim 7, characterized in that it outputs the filtering result based on the score.

9. The image generation unit is controlled by the processor, The generated image is obtained from the input text by the image generation process, The search unit is controlled by the processor, The generated image is used as a query to search the image-text database for similar images and obtain the related text for the similar images. The image generation assistance unit is controlled by the processor, Extracting multiple keywords from the aforementioned related texts, The image generation and search device according to claim 2, characterized in that the keyword selected by the user from a plurality of the aforementioned keywords is added to the input text and the generated image is obtained again by the image generation process.

10. The image generation unit is controlled by the processor, The generated image is obtained from the initial text. The image generation assistance unit is controlled by the processor, The system retrieves the modified text entered by the user, extracts existing elements and newly added elements from the initial text, The detected region obtained by detecting the existing element object from the generated image is reflected in the mask image as a retained region. The search unit is controlled by the processor, The modified text is used as a query to search for similar images in the image text database. The image generation assistance unit is controlled by the processor, From the aforementioned similar images, the objects of the existing element and the additional element are detected, The mask image is updated according to the frequency of appearance of the additional elements. The relative size of the additional element with respect to the existing element is calculated, The image generation and search device according to claim 2, characterized in that it generates the mask image corresponding to the relative size.

11. The aforementioned image and text database is The image generation and search device according to claim 10, characterized in that it holds the generated mask image.

12. The image generation and search device according to claim 1, An input device for specifying search criteria, A display device for displaying search results, An image generation and search system characterized by having the following features.

13. The aforementioned display device is It has an operation screen for performing the image generation process and the search, The aforementioned operation screen is The image generation search system according to claim 12, characterized in that it includes a generation process visualization field that displays the image generation process visualized by the display unit.

14. The aforementioned operation screen is A text input field for inputting the input text via the input device, An image generation button for acquiring the generated image via the input device, A generated image display field for displaying the generated image, A search condition field for specifying the search conditions via the input device, An image search button for performing the search via the input device, An image search results field for displaying the aforementioned search results, The image generation and search system according to claim 13, further comprising the following:

15. An image generation step in which a processor obtains a new generated image from input text and input image through an image generation process, The processor performs a document registration step in which it calculates document features from the input document, The processor performs an image registration step in which it calculates image features from the input image and the generated image, A storage step in which the input text, input image, generated image, image features, text features, and input / output relationships in the image generation process are stored in the image text database as part of the image generation process, The processor calculates similarity from the text features and image features, and uses the similarity to search for similar images from the image-text database that are similar to the search target; The processor performs a display step of visualizing the image generation process stored in the image document database, An image generation and search method characterized by having the following features.

Images