Image generation and search device, image generation and search system, and image generation and search method

Abstract

To provide an image generation and search device which enables a user to easily find a desired image from many image generation results.SOLUTION: Similarity is calculated from a sentence feature quantity and an image feature quantity and is used to search for a similar image similar to a search object from an image sentence database, and an image generation process held in the image sentence database is visualized.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image generation and retrieval device, an image generation and retrieval system, and an image generation and retrieval method. [Background technology]

[0002] The use of artificial intelligence (AI) technology has not only made it possible to recognize images with a high degree of accuracy (see Non-Patent Document 1), but also led to rapid developments in methods for generating the images themselves.

[0003] Proposed in 2014, generative adversarial networks (GANs) are a model that consists of two neural networks, a discriminative network and a generative network, and generates images that are close to the real thing from the generative network by having the two networks compete with each other to learn training images.

[0004] In early 2021, research progressed on methods of using large-scale language models to control the output of image generation models (see Non-Patent Document 2), and by 2022, it became possible to generate high-quality images using a diffusion model, a method different from GANs (see Non-Patent Document 3).

[0005] Recent image generation models can generate images of complex concepts that match the contents of explanatory text entered in natural language. Image generation by AI can be used not only for content production, but also for collecting training data for the AI ​​itself. Since it is particularly difficult to collect footage of rare cases such as disasters and accidents, if generative models can be properly controlled to generate large amounts of images that suit the use case, the development of image recognition systems can be accelerated.

[0006] For example, the image generation device of Patent Document 1 is characterized by having an acquisition unit that acquires information about ingredients input by a user and randomly generated random number data, and a trained generation unit that uses the information about the ingredients and the random number data acquired by the acquisition unit as input and generates the food image using a trained model for generating food images, thereby easily providing food images that have room for creativity and can help create a wide variety of dishes.

[0007] In addition, the information processing device of Patent Document 2 easily generates image data according to a user's desires by having a computer execute the steps of displaying a plurality of first image data, accepting a selection of a plurality of second image data from the plurality of first image data in response to a user's operation, generating a plurality of third image data having characteristics in accordance with a combination of characteristics of the plurality of second image data, and switching between and displaying the plurality of third image data in units of a predetermined number of sheets in response to the user's operation. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2019-153223 A (Patent No. 6865705) [Patent Document 2] JP 2020-102041 A [Non-patent literature]

[0009] [Non-Patent Document 1] GLIPv2: Unifying Localization and VL Understanding [Non-Patent Document 2] Learning Transferable Visual Models From Natural Language Supervision [Non-Patent Document 3] High-Resolution Image Synthesis with Latent Diffusion Models Summary of the Invention [Problem to be solved by the invention]

[0010] In the image generation device of Patent Document 1, the user can input the names of ingredients in natural language to control the content of the image output by the image generation model. However, since the output image changes depending on the randomly generated random numbers, it takes time to check the results. Also, if the expected image is not generated, it is necessary to repeat the process many times or try different combinations of ingredient names through trial and error.

[0011] In the information processing device of Patent Document 2, a user selects a generated image, a generating vector that is the basis of the generated image is obtained, and a new image is generated using the generating vector, thereby generating an image as intended by the user. However, since the generated image must be judged by the user, it takes time to judge when a large number of images are generated. In addition, feedback to the generation process is in a vector format that is unreadable by the user, so it cannot be applied to image generation using natural language as input.

[0012] An object of the present invention is to provide an image generation and retrieval device that enables a user to easily find a desired image from a large number of image generation results. [Means for solving the problem]

[0013] An image generation and retrieval device of one embodiment of the present invention is an image generation and retrieval device having a processor, comprising: an image generation unit that acquires a new generated image through an image generation process from an input text and an input image input by the processor; a text registration unit that calculates text features from the input text by the processor; an image registration unit that calculates image features from the input image and the generated image by the processor; an image and text database that holds the input text, the input image, the generated image, the image features, the text features, and the input and output relationships in the image generation process as an image generation process; a search unit that calculates a similarity from the text features and the image features by the processor and searches the image and text database for similar images that are similar to the search target using the similarity; and a display unit that visualizes the image generation process held in the image and text database by the processor. Effect of the Invention

[0014] According to one aspect of the present invention, it is possible to provide an image generation and retrieval device that enables a user to easily find a desired image from a large number of image generation results. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image generation and search system. [Diagram 2] FIG. 2 is a block diagram showing the hardware configuration of the image generation and search system. [Diagram 3] FIG. 1 shows the structure of an image and text database. [Figure 4A] FIG. 11 is a diagram illustrating an image generation process. [Figure 4B] FIG. 11 is a diagram illustrating an image generation process. [Figure 4C] FIG. 11 is a diagram illustrating an image generation process. [Diagram 5] 13 is a flowchart showing an image generation and database construction process. [Figure 6A] FIG. 13 is a diagram illustrating an image search process. [Figure 6B]FIG. 13 is a diagram illustrating an image search process. [Figure 7] 13 is a flowchart showing an image search process. [Figure 8] FIG. 1 is a diagram illustrating visualization of the image generation process. [Figure 9] 13 is a flowchart showing a visualization process of an image generation process. [Figure 10] FIG. 13 is a diagram illustrating filtering of image generation results by similar image search. [Figure 11] 13 is a flowchart showing a filtering process for image generation results by similar image search. [Figure 12] FIG. 13 is a diagram illustrating filtering of an image generation result by image recognition. [Figure 13] 11 is a flowchart showing a filtering process for an image generation result by image recognition. [Figure 14] FIG. 13 is a diagram illustrating assistance in inputting text by using similar image search. [Figure 15] 13 is a flowchart showing a text input assistance process using similar image search. [Figure 16] FIG. 13 is a diagram illustrating assistance in inputting a mask image by using a similar image search. [Figure 17] 13 is a flowchart showing a mask image input assistance process using similar image search. [Figure 18] FIG. 13 is a diagram showing an example of a screen of the image generation and search system. [Figure 19] FIG. 1 is a sequence diagram showing the overall processing of the image generation and search system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. The embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention. The same reference symbols are used for common components in each drawing. EXAMPLES

[0017] The image generation / search device 104 of the first embodiment automatically generates an image according to the content of the text based on the input text or image. It also extracts features from the input or generated text / image, and builds an image / text database 110 for similar image search by combining them with information on the input / output relationship of image generation. By using the database search function, a user can check the process of past image generation and efficiently find a desired image from a large number of generated images.

[0018] FIG. 1 is a block diagram showing an example of the configuration of an image generation and search system 100 according to a first embodiment.

[0019] Possible use cases for the image generation and search system 100 include, but are not limited to, content production and data collection for machine learning. Each component will be described below.

[0020] The image generation and search system 100 automatically generates an image according to the contents of a sentence from a sentence or image input by a user, and registers the image in an image and sentence database 110. The image generation and search system 100 is composed of an image storage device 101, an input device 102, a display device 103, and an image generation and search device 104. The image storage device 101 is a storage medium for storing still image and video data or associated bibliographic information, and is composed of a hard disk drive built into a computer, or a storage system connected via a network such as a NAS (Network Attached Storage) or a SAN (Storage Area Network). The image storage device 101 may also be a cache memory for temporarily storing data continuously input from a photographing device.

[0021] The input device 102 is an input interface, such as a mouse, keyboard, or touch device, for transmitting user operations to the image generation / search device 104. It may also be a voice input interface that allows input of text by voice recognition. The display device 103 is an output interface, such as a liquid crystal display, and is used for displaying search results of the image generation / search device 104 and for interactive operations with the user.

[0022] The image generation / search device 104 is a device that performs an image generation process to generate an image that matches the content of the text from text or images input by the user, a registration process to create a database of the input / generated images / text, and a search / display process to process the generated results and present them to the user.

[0023] The image generation / search device 104 has a text input unit 105 , an image input unit 106 , an image generation unit 107 , a text registration unit 108 , an image registration unit 109 , an image / text database 110 , a search unit 111 , a display unit 112 , and an image generation assistant unit 113 .

[0024] The image generation and database registration process will be described below. Details of the process will also be explained in the flowchart of FIG.

[0025] In the image generation process, an image is automatically generated based on the content of the text from the text input by the user via the input device 102. It is also possible to input an existing image recorded in the image storage device 101 and modify it based on the content of the text. In the database registration process, feature amounts for similar image search are extracted from the input text / image and the generated image, and are registered in the image / text database 110.

[0026] A feature is a multi-dimensional vector-formatted numerical data, and by calculating the cosine similarity between two vectors, it is possible to determine the similarity of the data from which the feature was extracted. By using a model trained on pairs of images and text for feature extraction, it is possible to calculate similarities between images, images and text, and text and text using a similar procedure. In addition, in the database registration process, the input / output relationship of the image generation process is registered in the image / text database 110. This makes it possible to present the process of past image generation processes.

[0027] The text input unit 105 accepts text input by the user using the input device 102, and converts it into a data format used within the image generation / retrieval device 104. The image generation method targeted by the present invention can basically accept text in a natural language as input, but there are often restrictions on the type of language and the length of the text.

[0028] In addition, a set of predetermined words may be accepted instead of a natural language. Furthermore, words may be emphasized or weakened according to specific description rules. For this reason, the text input unit 105 converts the input text according to the image generation model used in the image generation unit 107. For example, a Japanese text is translated into an English text. When multiple machine learning models are used in the image generation unit 107, multiple converted texts converted according to the respective constraints are output.

[0029] The image input unit 106 receives input of still image data or video data from the image storage device 101, and converts it into a data format used within the image generation and search device 104. For example, if the data received by the image input unit 106 is video data, the image input unit 106 performs a video decoding process to break it down into frames (still image data format). In addition, if the data is stroke information input using a mouse or touch device, it draws it on an image.

[0030] The image generation unit 107 uses an image generation model to automatically generate an image based on the content of one or more sentences. The image generation model uses an image generation algorithm such as that described in Non-Patent Document 3, and uses model parameters obtained by training a large number of images and their descriptions through deep learning. Therefore, the generated image changes significantly not only depending on the input sentence, but also on the image generation algorithm and learning data. In addition, in an algorithm in which the initial value is given by random noise, the generated image also changes depending on the random number seed.

[0031] Depending on the image generation algorithm, it is possible to input an image and change the image to match the sentence, or input a mask image to change only a part of the image. It is also possible to input two or more sentences and generate an image of an intermediate concept, or input a negative sentence and generate an image that does not include the elements of the sentence. In this way, the image generation unit 107 can generate a wide variety of images according to the input. In addition, by using multiple image generation models, it is possible to generate a large number of different images at once from the conditions input by the user.

[0032] The text registration unit 108 extracts features from the input text acquired by the text input unit 105 and used to generate an image. Features are usually given as vector data of a fixed length, and the similarity between the two original data can be found by calculating the Euclidean distance between two vectors, cosine similarity, etc. In addition, by using the language / image feature extraction model described in Non-Patent Document 2, features that allow comparison of similarity can be obtained from text and images.

[0033] Image registration unit 109 extracts features from the input image acquired by image input unit 106 or the generated image generated by image generation unit 107. In the image registration unit 109 of the present invention, features are extracted from the image using a feature extraction model similar to that of sentence registration unit 108.

[0034] The image and text database 110 holds input text and images or generated images and their features, as well as information on the input and output relationships of the image generation process. In response to queries from each section of the image generation and search device 104, the image and text database 110 can search for registered data that meets given conditions and read data with a specified ID. It can also output registered data similar to a query by using features extracted from images and text. The structure of the image and text database 110 will be described in detail later with reference to FIG. 3.

[0035] The above is the operation of each unit in the image generation and database registration processing of the image generation / search device 104. Next, a description will be given of the operation of each unit in the search / display processing of the image generation / search device 104. Details of the processing will also be described in the flowchart of FIG.

[0036] In the search and display process, search conditions specified by the user through the input device 102 are used to search for images that match the search conditions from the image and text database, and the information is presented on the display device 103. For example, the database can be searched using a search query given in the form of text or an image, and images registered in the database can be sorted and displayed in order of similarity.

[0037] The text used in image generation may be used as the search query, or the image resulting from image generation may be used as the query. This allows the user to obtain a desired image not only from the image generated in the immediately preceding image generation procedure, but also from images generated in the past. The search and display process can also visualize the input / output relationship of image generation registered in the image and text database 110. This allows the user to generate a new image by referring to the past generation process.

[0038] The search unit 111 retrieves image and text data from the image and text database 110 using specified search conditions. When a query is given as a conditional expression, registered data that matches the conditions is returned, and when a query is given as vector data, similarity between vectors is calculated and data sorted according to the similarity is returned. The search unit 111 can also process the images and text retrieved from the image and text database 110 by necessary analysis processing so that the user can obtain the information he or she needs.

[0039] The display unit 112 displays the data acquired from the image / text database 110 on the display device 103. For example, the generated image and its similar images may be displayed side by side, or a graph showing the input / output relationship in the image generation process may be displayed.

[0040] The image generation assistance unit 113 uses the results of the search unit 111 to assist the user in inputting image generation. For example, from similar images acquired by the search unit 111, one image specified by the user can be used as an initial image for generating a new image, or a sentence used in the past can be used again as an input sentence. Also, for example, when multiple image generation models are used, only the image generation model that has output many results that the user judges to be desirable can be activated to generate a new image.

[0041] The above is the operation of each part in the search and display process of the image generation and search device 104. Note that the image generation and registration process and the search and display process of the image generation and search device 104 are processes that are repeatedly performed in response to user instructions, and the contents of the image and text database 110 are added and updated successively, and the processing contents of each part that uses the registered data change accordingly. Also, if exclusive control of database updates is performed appropriately, multiple users can access and use it simultaneously.

[0042] 2 is a block diagram showing an example of the hardware configuration of the image generation and retrieval system 100 of this embodiment. The image generation and retrieval device 104 includes a processor 201 and a storage device 202 that are connected to each other. The storage device 202 is configured with any type of storage medium. The storage device 202 is configured with a combination of storage devices such as semiconductor memories and hard disk drives.

[0043] 1, functional units such as the text input unit 105, the image input unit 106, the image generation unit 107, and the text registration unit 108 are realized by the processor 201 executing a processing program 203 stored in the storage device 202. In other words, the processing performed by each functional unit is executed by the processor 201 based on the processing program 203.

[0044] The data of the image / text database 110 is stored in a storage device 202. When the image generation / search system 100 is configured with a plurality of devices for the purpose of distributing the processing load, the device having the image / text database 110 and the device executing the processing program 203 may be physically different devices connected via a network, or the processing program 203 may be executed simultaneously in a plurality of devices as long as the consistency of the data recorded in the image / text database 110 can be maintained.

[0045] The image generation and retrieval device 104 further includes a network interface device (NIC) 204 connected to the processor 201. It is assumed that the image storage device 101 is a NAS or a SAN connected to the image generation and retrieval device 104 via the network interface device 204. Note that the image storage device 101 may be included in the storage device 202.

[0046] Fig. 3 is an explanatory diagram showing an example of the structure and data of the image / text database 110 of this embodiment. Note that in this embodiment, the information used by the system may be expressed in any data structure independent of the data structure. Fig. 3 shows an example of a table format, but the information can be stored in any data structure appropriately selected from, for example, a table, a list, a database, or a queue.

[0047] The image and text database 110 includes, for example, an image and text information table 300. The table configuration and the field configuration of each table in FIG. 3 are merely examples, and tables and fields may be added depending on the application, for example. Furthermore, the table configuration may be changed as long as similar information is held. For example, the image and text information table 300 may be divided into an image information table and a text information table, or information relating to input and output for image generation may be managed in a separate table.

[0048] The image / text information table 300 is made up of an ID field 301 , a type field 302 , an image field 303 , a text field 304 , an input text ID field 305 , an input image ID field 306 , an input mask ID field 307 , and a feature field 308 .

[0049] The ID field 301 holds an identification number of image / text information. The type field 302 holds the type of information. The types of information are, for example, text, image, mask image, etc. Here, a mask image is an image in which information on an area that changes an input image during image generation is recorded. The image field 303 holds binary data of the image if the type is an image. If the binary data of the image can be accessed, it may be a file path on file storage.

[0050] The text field 304 holds the text character string when the type is text. The input text ID field 305 holds the ID managed in the image / text information table 300 for the input text used in image generation. The input image ID field 306 holds the ID managed in the image / text information table 300 for the input image used in image generation. The input mask ID field 307 holds the ID managed in the image / text information table 300 for the input mask image used in image generation. The feature field 308 holds a numerical vector representing a feature extracted from an image or text.

[0051] The image / text information table may add fields as necessary. For example, the time when the data was registered, information on the model used for image generation, various parameters used for image generation (e.g., random number seed), user ratings of the generated image, etc. may be added. In addition, when using an image generation model that accepts multiple input texts or input images, the input text ID field 305, input image ID field 306, and input mask ID field 307 may be configured to hold multiple IDs. Multiple fields may also be prepared for the feature field 308, and the results of calculations using multiple feature extraction methods may be stored, allowing the user to select which feature to use when searching.

[0052] 4A, 4B, and 4C are diagrams for explaining an overview of the image generation process of the image generation unit 107 of this embodiment. The image generation process is a process for automatically generating an image according to the contents of an input sentence, and can use a known algorithm such as that shown in Non-Patent Document 3. The image generation process of the image generation unit 107 of this embodiment mainly uses three types of input / output patterns.

[0053] In image generation from a sentence, one or more images 402 are generated from an input sentence 401 according to the content of the sentence (see FIG. 4A).

[0054] In image generation from text and image, the input image 403 is modified in accordance with the content of the text (see FIG. 4B).

[0055] In image generation from text, image, and mask, corrections are made to the area of ​​the input image specified by the mask image 404 (see Fig. 4C). In Fig. 4C, the black-filled areas of the mask image 404 retain the information of the input image, and the white-filled areas generate an image according to the input text. The mask image may be given as a binary value indicating whether or not corrections are made, or as a continuous value indicating the strength of the corrections.

[0056] Although three types of patterns have been described above, the image generation process of the image generation unit 107 of this embodiment is not limited to these, and a derived image generation algorithm with a different number of inputs and outputs may be used by adding a field to the image / text information table 300. For example, it may be possible to input a plurality of texts and generate an image of an intermediate concept. In addition, it may be possible to respond to a process (OutPainting) of drawing outside the frame of the input image by resizing or trimming the input image and inputting it to the image generation model.

[0057] Fig. 5 is a diagram showing the process flow of image generation and database registration. The image generation and search system 100 of this embodiment performs image generation when triggered by a user inputting text for image generation, and automatically performs database registration unless otherwise specified. Each step of Fig. 5 will be explained below.

[0058] The text input unit 105 receives text-like information from the input device 102 and converts it into text data that can be used within the system as necessary (S501). The text registration unit 108 calculates features from the text data acquired in step S501 and registers them in the image and text database 110 (S502).

[0059] If the initial image is used as the input for the image generation process, the image generation / retrieval device 104 executes step S504, otherwise executes step S506 (S503). The image input unit 106 acquires the initial image to be the input for the image generation process from the image storage device 101, and converts it into image data that can be used within the system as necessary (S504).

[0060] The image registration unit 109 calculates image features from the initial image data and registers them in the image / text database 110 (S505). The image generation assistant unit 113 receives image generation conditions from the input device 102 (S506). The image generation conditions are, for example, parameters such as the input pattern for the image generation process described in the description of FIG. 4, the type of generation model, the random number seed, and the number of images to be output. Furthermore, when a mask image is used in the generation process, information on the mask area is received from the input device 102 and the mask image is generated.

[0061] The image generation / retrieval device 104 executes step S508 if a mask image is used in the image generation process, and executes step S510 if not (S507).

[0062] Image registration unit 109 calculates features from the mask image generated in step S506 and registers them in image / text database 110 (S508). Image generation unit 107 generates an image from the text input in step S501 according to the conditions input in step S506 (S509). Depending on the specified conditions, the initial image input in step S504 and the mask image generated in step S506 are added to the input for image generation processing.

[0063] The image registration unit 109 calculates features from the generated image obtained in step S509, and registers the calculated features in the image / text database 110 (S510). When registering the generated image, information about the input data used in the generation process is also recorded.

[0064] 6A and 6B are diagrams for explaining an overview of similar image search processing by the search unit 111 of this embodiment. The image generation and search device 104 of this embodiment extracts mutually comparable feature amounts from images and text, and stores them in the image and text database 110. Therefore, by comparing the feature amounts, it is possible to obtain similar data from the image and text database 110 using an image or text as a query.

[0065] In the search process, (a) image search from a query sentence, (b) image search from a query image, (c) text search from a query sentence, and (d) text search from a query image can be performed. Figure 6A shows an example of (a) for acquiring similar images. Figure 6B shows an example of (b) for acquiring similar images.

[0066] As shown in Fig. 6A, in image search from a query sentence, features are extracted from the input query sentence and used as query features. The data registered in the image / sentence database 110 that are of the image type are targeted, and the similarity between the registered features and the query features is calculated. The features are multidimensional vector data, and the similarity between two vectors can be calculated using, for example, cosine similarity.

[0067] Search results 601 are given by IDs managed in the image / text information table 300 and similarities obtained by vector calculations. When performing image search from a query text, emphasis is placed on whether the concept of the text is included, so image features of search results tend to differ from each other.

[0068] As shown in Fig. 6B, the procedure is similar for image search from a query image, where query features are calculated from the input query image and the image and text database 110 is searched. When a query image is used, the features include various image features that cannot be expressed in text, so images with similar atmosphere and composition tend to appear in the search results. On the other hand, images that do not include the intended concept tend to appear in the search results.

[0069] 7 is a diagram showing a processing flow of similar image search by the search unit 111 of this embodiment. Each step in FIG. 7 will be described below.

[0070] The search unit 111 executes step S702 if the search query is a sentence, executes step S703 if the search query is an image, and executes step S704 if the search query is specified by a database ID (S701).

[0071] If there is a search narrowing condition, the search unit 111 executes step S706, otherwise executes step S707 (S705). The search unit 111 extracts features from the query text (S702). The feature extraction process is performed in the same manner as the text registration unit 108.

[0072] The search unit 111 extracts features from the query image (S703). The feature extraction process is performed in the same manner as the image registration unit 109. The search unit 111 acquires the features of the specified ID from the image / text database 110 (S704).

[0073] The search unit 111 uses the specified narrowing conditions to narrow down the records registered in the image / text database 110 to which similarity calculation is to be performed (S706). For example, if the narrowing conditions specify that the type is only images, the search results that are calculated and output will be only images.

[0074] The search unit 111 calculates the similarity between all records to be subjected to the similarity calculation and the feature amount acquired in step S702, step S703, or step S704 (S707). The cosine similarity of two vectors can be used for the similarity calculation. Also, a known approximate nearest neighbor search method may be applied, which searches only for records with high similarity at high speed without calculating the similarity of all records by collecting and saving data with high similarity in advance.

[0075] The search unit 111 sorts the search results using the similarity and outputs the specified number of results (S708). In similar image search, the search results are usually output in order of decreasing similarity, but depending on the purpose, the results may be output in order of decreasing similarity or an upper or lower limit may be set for the similarity. Also, by thinning out the search results so that the similarity between the search results is below a certain level, a more diverse search result may be output.

[0076] 8 is a diagram for explaining an overview of visualization of the image generation process on the display unit 112 of this embodiment. Since the image / text database 110 of this embodiment holds information on input / output relationships in the image generation process, by visualizing this information, the user can learn effective procedures from past image generation.

[0077] In Fig. 8, the generation process is visualized as a directed graph consisting of nodes and edges. Each node represents a record registered in the image and text database 110, and displays a thumbnail of an image or a text depending on the type. The edges represent input / output relationships, with the node connected to the start point of the arrow being the input data, and the node connected to the end point being the output generated image. For example, the process by which image 802 is generated from text 801 is represented by edge 803. The directed graph makes it possible to visually grasp the generation process of any image.

[0078] For example, it can be seen that text 801 has been used in many image generation processes, and image 806 has been generated using text 803 and mask 805 in comparison to image 804, which was obtained through several image generation processes. In graph visualization, visibility can become poor as the amount of data increases, but it is possible to apply commonly conceived simplified display methods, such as displaying only the area around a node of interest or displaying all generation processes under the same conditions together.

[0079] In addition, since the records represented by each node have features that allow the similarity to be calculated, it is possible to control the arrangement of nodes with similarities in close positions. For example, since sentences 801 and 803 have a high similarity, the drawing of the graph may be controlled by giving them a virtual edge 808.

[0080] 9 is a diagram showing a processing flow for visualizing the image generation process of the display unit 112 of this embodiment. Each step in FIG. 9 will be described below.

[0081] The search unit 111 acquires information on a record to be rendered designated by a user from the image / text database 110 (S901).

[0082] The display unit 112 adds a node to the graph using the information of the record to be drawn acquired in step S901 (S902).

[0083] The display unit 112 executes steps S904 to S906 for the input record in the generation process held by the rendering target record acquired in step S901 (S903). The input record here refers to a record of the ID held in the input text ID field 305, the input image ID field 306, and the input mask ID field 307 of the image / text information table 300.

[0084] If the input record has already been rendered in a graph, the display unit 112 executes step S906; if not, it executes step S905 (S904).

[0085] The image generating / retrieving device 104 performs the process flow of FIG. 9 with the input record as the record to be rendered (step S905).

[0086] The display unit 112 adds an edge between the node of the record to be rendered and the node of the input record (S906). The edge is a directed edge from the node of the input record to the node of the record to be rendered.

[0087] When the process for all input records is completed, the display unit 112 executes step S908 (S907). If the type of the record to be rendered is text and a node of a text record with high similarity exists, the display unit 112 executes step S909; otherwise, the display unit 112 executes step S909 (S908).

[0088] The display unit 112 adds a virtual edge between the node of the record to be drawn and the node of the sentence record with high similarity (S909). The virtual edge is added to control the drawing position of the node, and may or may not be displayed on the screen presented to the user.

[0089] The display unit 113 optimizes the arrangement of the nodes according to the connection status of the edges (S910). A known graph drawing method can be used for the optimization. For example, a method can be used that determines, by repeated calculation, an arrangement in which the basic repulsive forces between the nodes and the attractive forces between the nodes connected by edges are balanced. EXAMPLES

[0090] The image generation / search device 104 of the first embodiment enables a user to efficiently find a desired image by searching for generated images stored in an image / text database using text or images, and by visualizing the image generation process. However, when generating a new image, the user needs to explicitly perform an image search or check a visualized graph of the generation process.

[0091] The image generation / search device 104 of the second embodiment automatically analyzes a large number of images generated under predetermined conditions, and preferentially presents images that the user expects. This allows efficient image generation even when generating images under new conditions.

[0092] 10 is a diagram for explaining the filtering process of the image generation result based on the similarity in the embodiment 2. The image generation unit 107 can generate a large number of images in response to one image generation request from the user by changing the image generation model and the parameters of the image generation.

[0093] The image generation / search device 104 of the second embodiment performs similarity calculations on the generation results 1001 using a pre-set query set 1002, calculates a score, and outputs images with scores equal to or greater than a predetermined threshold as filtering results 1003.

[0094] The query set 1002 can be given as images or text, and can be added interactively by the user. The score of the image to be evaluated is calculated, for example, by calculating the similarity between each query and the image to be evaluated, weighting it as necessary, and adding it up. By setting a negative value as the weight of the query, it is also possible to increase the score of an image with a low similarity to the query. For example, by collecting generated images with a low similarity to the query, a wide variety of images can be obtained.

[0095] Fig. 11 is a diagram showing a processing flow of filtering the image generation result based on the similarity in the embodiment 2. Each step in Fig. 11 will be described below.

[0096] The image generation unit 107 generates a predetermined number of images using a predetermined combination of image generation models and image generation parameters (S1101).

[0097] The search unit 111 acquires filtering conditions from the input device 102 (S1102). The filtering conditions include queries using one or more images or text, weights for each query, score thresholds, and the like.

[0098] The search unit 111 executes steps S1104 to S1105 for each query specified by the filtering condition acquired in step S1102 (S1103). The search unit 111 performs a similar image search process using the query image or query sentence on the generated image set generated in step S1101, and acquires search results (S1104).

[0099] The search unit 111 adds the similarity with the query acquired in step S1104 to the score of each generated image (S1105). If a weight is assigned to the query in the filtering condition acquired in step S1102, the search unit 111 adds the similarity with the query after multiplying the weight.

[0100] When the search unit 111 has finished processing all queries, it executes step S1107 (S1106). The search unit 111 sorts the generated images in descending order of total score and outputs a predetermined number of them (S1107). If a score threshold is set in the filtering conditions acquired in step S1102, the search unit 111 outputs generated images that are equal to or greater than the threshold.

[0101] 12 is a diagram for explaining a filtering process of an image generation result by image recognition in the embodiment 2. The image generation unit 107 can automatically generate an image according to the contents of an input sentence, but depending on an image generation model or image generation parameters, an image that does not necessarily include all concepts included in the input sentence may be generated.

[0102] The image generation / retrieval device 104 of the second embodiment automatically verifies whether each concept included in the input sentence is also included in the generated image by image recognition processing. In the example of Fig. 12, an object is detected from the generated image by image recognition processing, and a score is calculated according to whether the object "cat", "apple", and "table" included in the input sentence are included, and filtering processing is performed.

[0103] Fig. 13 is a diagram showing a processing flow of filtering the image generation result by image recognition in the embodiment 2. Each step in Fig. 13 will be described below.

[0104] The image generation unit 107 generates a predetermined number of images using an image generation model and image generation parameters that are a predetermined combination with the input text (S1301).

[0105] The search unit 111 acquires filtering conditions from the input device 102 (S1302). The filtering conditions are an object detection algorithm used for image recognition and its parameters. A known algorithm shown in Non-Patent Document 1 can be used as the object detection algorithm.

[0106] The search unit 111 extracts a word list of objects that can be detected by the object detection algorithm specified in step S1302 from the input sentence used for image generation in step S1301 (S1303).

[0107] The search unit 111 executes steps S1305 to S1306 for each query specified by the filtering condition acquired in step S1302 (S1304).

[0108] The search unit 111 executes an object detection process for each generated image generated in step S1301 (S1305). The search unit 111 calculates a score based on the degree of match between the list of objects detected in step S1304 and the word list acquired in step S1303, and adds the score to the total score of each generated image (S1306).

[0109] When the search unit 111 has finished processing all queries, it executes step S1308 (S1307). The search unit 111 sorts the generated images in descending order of total score and outputs a predetermined number of them (S1308). If a score threshold is set in the filtering conditions acquired in step S1302, the search unit 111 outputs generated images that are equal to or greater than the threshold.

[0110] 11 and 12 show an example in which the degree of matching between the contents of the input text and the generated image is determined using an object detection algorithm, but the image generation / search device 104 of the second embodiment can use any image recognition algorithm as long as it can determine the degree of matching between the contents of the input text and the generated image.

[0111] For example, the degree of match may be determined based on the similarity between the text generated using an image description generation algorithm and the input text. Even when an object detection algorithm is used, the degree of match may be calculated by taking into account the positional relationship of the object. For example, based on the relationship between "cat" and "on the table," the score of a generated image in which "cat" is located on "table" in the object detection results may be increased. EXAMPLES

[0112] The image generation / retrieval device 104 of the second embodiment can efficiently present a user-desired image from a large number of images generated under predetermined conditions through image analysis processing. On the other hand, the user needs to adjust the image generation conditions through trial and error.

[0113] The image generation / search device 104 of the third embodiment can search for previously generated images stored in the image / document database 110, thereby presenting candidates for sentences and mask images to be input for image generation to the user.

[0114] 14 is a diagram for explaining text input assistance by image search in the third embodiment. A similar image generated in the past is searched for in the image / text database 110 using a generated image obtained using an input text as a query. Since the image / text database 110 holds information on the input / output process of the image generation process, the input text used in the image generation process can be obtained from each image in the search results.

[0115] In the image generation assistant unit 113 of the third embodiment, keywords used for image generation can be extracted by analyzing text obtained from similar images. Keyword extraction may be performed by simply listing the words that have appeared, or by calculating the score of the words using any statistical index and changing the display order according to the score. For example, the term frequency (TF) may be used as the index, or the statistic TF-IDF, which is the product of TF and inverse document frequency (IDF), may be used as the index. Here, TF is the frequency of a word included in the similar image search result, and IDF is the reciprocal of the number of texts including the relevant word from all records in the image and text database 110. Using TF-IDF, words that appear more frequently than usual in the similar image search result can be extracted.

[0116] Fig. 15 is a diagram showing a processing flow of text input assistance by image search in the embodiment 3. Each step in Fig. 15 will be described below.

[0117] The image generating unit 107 generates an image from an input sentence (S1501). Step S1501 is equivalent to the image generating process described with reference to FIG. If the image generation assistant unit 113 determines that the user's desired image has been obtained, it ends the process, and if not, it executes step S1503 (S1502).

[0118] The image search unit 111 searches for similar images from the image / text database 110 using the generated image acquired in step S1501 as a query (S1503). Step S1503 is equivalent to the image search process described with reference to FIG.

[0119] The image generation assistant unit 113 executes step S1505 for each similar image acquired in step S1503 (S1504).

[0120] The image search unit 111 acquires the related sentences of the similar image from the image / sentence database 110 (S1505). The related sentences are, for example, data held in the sentence field 304 of the record corresponding to the ID held in the input sentence ID field 305 of the record of the similar image.

[0121] When the processing for all similar images is completed, the image generation assistant unit 113 executes step S1507 (S1506). The image generation assistant unit 113 extracts a set of keywords from the related sentences acquired in step S1505 (S1507). As described in the explanation of FIG. 14, the method of extracting keywords may be a simple enumeration of words, or may be a result of scoring using a statistical index such as TF or TF-IDF.

[0122] The image generation assistant unit 113 adds to the input sentence the keyword selected by the user through the input device 102 from the keyword set extracted in step S1507 (S1508). At this time, the keyword may simply be added to the end of the sentence, or a sentence obtained using a known sentence generation algorithm that generates a sentence from a keyword may be added.

[0123] Fig. 16 is a diagram for explaining mask input assistance by image search in embodiment 3. As explained in Fig. 4C, the image generation and search device 104 of the present invention can generate an image in which a part of the input image is corrected by inputting a mask image.

[0124] In the image generation assistant unit 113 of the third embodiment, when generating an image with a revised sentence by adding a new additional element while leaving the existing elements contained in the initial sentence as they are, the image generation assistant unit 113 searches for similar images from the image / sentence database 110 using the revised sentence and calculates a frequency map showing in which area of ​​the image the additional element is likely to appear. From this frequency map and the area of ​​the existing elements in the image generated from the initial sentence, a candidate area for placing the additional element is obtained. In addition, the relative size of the additional element to the existing elements is calculated, and one or more areas in which the relative size can be placed are selected from within the candidate area to generate a candidate mask image.

[0125] Fig. 17 is a diagram showing a process flow of mask input assistance by image search in the embodiment 3. Each step in Fig. 17 will be described below.

[0126] The image generation unit 107 generates an image from the initial sentence (S1701). Step S1701 is equivalent to the image generation process described in Fig. 5. The image generation assistant unit 113 analyzes the corrected sentence input by the user, and extracts existing elements that were also included in the initial sentence and newly added additional elements (S1702).

[0127] The image generation assistant unit 113 detects the object of the retained element extracted in step S1702 from the generated image acquired in step S1701 (S1703). The image generation assistant unit 113 reflects the object area detected in step S1703 as a retained area in the mask image (S1704). Here, the retained area is an area that is not modified by the image generation process, and is represented as a black-filled area in the mask image in FIG.

[0128] The search unit 111 searches the image / sentence database 110 for similar images using the corrected sentence obtained in step S1702 as a query (S1705). The image generation assistant unit 113 detects objects of the existing elements and the added elements extracted in step S1702 from each similar image obtained in step S1705 (S1706).

[0129] The image generation assistant unit 113 creates an appearance frequency map by aggregating the appearance regions of the added elements detected in step S1706, and updates the mask image so that the locations with high appearance frequencies are treated as correction regions (S1707). Here, the correction region is the region that is corrected by the image generation process, and is represented as a region filled with white in the mask image in FIG.

[0130] The image generation assistant unit 113 calculates the relative size of the additional element with respect to the existing element from the object regions of the existing element and the additional element detected in step S1706 (S1708).

[0131] The image generation assistant unit 113 selects one or more areas from the modified area of ​​the mask image obtained in step S1707 that contain the relative size of the added element calculated in step S1708, and generates a mask image with the selected areas as modified areas (S1709).

[0132] The image generation / retrieval device 104 according to the first to third embodiments described above can generate a large number of images according to the contents of the input text, and the user can efficiently find the image he or she desires from the generated images. In addition, in the procedure for generating a new image, the generated image can be filtered and displayed by image analysis processing, and candidates for text and mask images used in image generation can be presented, thereby enabling efficient generation.

[0133] FIG. 18 is a diagram showing an example of the configuration of an operation screen for performing image generation and image search in the image generation / search device 104 according to the first to third embodiments.

[0134] The image generation / search device 104 displays the processing results on the display device 103. The user communicates operation information to the image generation / search device 104 using a mouse cursor 1801 displayed on the screen by the input device 102. The screen is composed of a text input field 1802, an image generation button 1803, a generated image display field 1804, a generation process visualization field 1805, a search condition field 1806, an image search button 1807, an image search result field 1808, a text candidate field 1809, an add text button 1810, a mask candidate field 1811, and a mask setting button 1812. The screen configuration example is just an example, and the screen may be configured by freely arranging these elements.

[0135] FIG. 19 is a sequence diagram showing the process of image generation and image search in the image generation / search device 104 according to the first to third embodiments.

[0136] Fig. 19 specifically shows the processing sequence between the user 1900, the image storage device 101, the computer 1920, and the image and text database 110 in each process of the image generation and retrieval system 100 described above. The sequence in Fig. 19 is roughly divided into the input assistance process described in the third embodiment, the image generation and database registration process described in the first embodiment, the output control process, and the output control process described in the second embodiment, and these sequences are repeatedly executed in response to a user request. The computer 1920 is a computer that realizes the image generation and retrieval device 104. Each step in Fig. 19 will be described below.

[0137] When a user 1900 issues an image generation request (S1901), a series of processes related to image generation is started in a computer 1920. When an input image is to be used for image generation, the computer 1920 requests the input image from the image storage device 101 (S1902), and the image storage device 101 returns the input image (S1903).

[0138] The computer 1920 sends a similar image search request using the input image as a query to the image and document database 110 (S1904), and the image and document database 110 returns the search results (S1905). The computer 1920 estimates additional sentence candidates from sentences acquired from the search results (S1906) and presents the additional sentence candidates to the user 1900 (S1907). The user corrects the input sentence using the presented candidates and inputs it to the computer 1920 (S1908). The computer 1920 estimates mask candidates from the corrected sentences and the image search results (S1909), and presents the mask candidates to the user 1900 (S1910). The user 1900 inputs the mask image to the computer 1920 with reference to the presented mask image (S1911).

[0139] The computer 1900 generates an image using the input text, image, and mask image (S1912). The computer 1900 extracts features from the text, image, and mask image used in the generation and the generated image (S1913), and registers them in the image and text database 110 (S1914). The image and text database 110 returns the ID of the record that has been registered (S1915). The computer 1920 obtains record information from the image and text database 110 using the ID (S1917), performs filtering and visualization processing (S1918), and presents the results to the user (S1919).

[0140] According to the above embodiment, it is possible to present information that makes it easier to find images. In addition, it is possible to present appropriate auxiliary information to the user when redoing image generation. It is believed that the effect of the above embodiment will become more pronounced with the emergence of a method that can generate a large number of images with diverse concepts and variations using natural language.

[0141] More specifically, according to the above embodiment, it is possible to narrow down and present images that match the user's intention from a large number of images generated by text input. Also, it is possible to assist input for image generation by searching generated images stored in an image / text database, extracting conditions such as text candidates to be input for new image generation, and presenting them to the user. Furthermore, it is possible to visualize the input / output relationship of image generation stored in the database and present it to the user.

[0142] The above describes the embodiments of the present invention. Note that the present invention is not limited to the above-mentioned embodiments, and various modifications are included. For example, the above-mentioned embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0143] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0144] Furthermore, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0145] 100 Image generation and search system 101 Image storage device 102 Input Device 103 Display device 104 Image generation and retrieval device 105 Text input section 106 Image input unit 107 Image Generation Unit 108 Text Registration Section 109 Image Registration Section 110 Image and text database 111 Search Department 112 Display section 113 Image Generation Assistant

Claims

1. An image generation and retrieval device having a processor, an image generation unit that acquires a new generated image by image generation processing from the input sentence and the input image input by the processor; a sentence registration unit that calculates sentence features from the input sentence by the processor; an image registration unit that calculates image features from the input image and the generated image by the processor; an image and text database that holds the input text, the input image, the generated image, the image feature amount, the text feature amount, and an input / output relationship in the image generation process as an image generation process; a search unit that calculates a similarity between the text feature amount and the image feature amount and searches the image / text database for a similar image that is similar to a search target using the similarity; a display unit that visualizes the image generation process stored in the image and text database by the processor; An image generation and retrieval device comprising:

2. a sentence input unit configured to accept the input sentence by the processor; an image input unit configured to receive the input image by the processor; an image generation assistant unit that assists the image generation process of the generated image by the processor; 2. The image generating and retrieving device according to claim 1, further comprising:

3. The search unit, by the processor, searching the image-text database using a search query provided in the input text or the input image; 2. The image generating and searching device according to claim 1, wherein the searched similar images are rearranged according to the degree of similarity.

4. The display unit, by the processor, 2. The image generation and retrieval device according to claim 1, wherein the image generation process is visualized as a graph that shows the input / output relationship by arranging the generated image and the similar images.

5. The image generation unit, by the processor, A predetermined number of the generated images are obtained as a generated image set; The search unit, by the processor, The image generation and search device according to claim 1, characterized in that a filtering process using the similarity is performed to search for similar images using a query image or a query sentence within the generated image set, and the search results are output as filtering results.

6. The search unit, by the processor, In the filtering process, the similarity is calculated using a query set that is set in advance for the generated image to calculate a score; The image generation and search device according to claim 5 , wherein the filtering result is output based on the score.

7. The image generation unit, by the processor, obtaining a predetermined number of the generated images from the input sentence; The search unit, by the processor, The image generation and search device according to claim 1, characterized in that a filtering process using image recognition is performed to recognize whether the concepts of words contained in the input sentence are contained in the generated image, and the recognition result is output as the filtering result.

8. The search unit, by the processor, performing an object detection process on the generated image; Calculating a score based on a degree of match between the object detection result in the object detection process and the word included in the input sentence; The image generation and retrieval device according to claim 7 , wherein the filtering result is output based on the score.

9. The image generation unit, by the processor, obtaining the generated image from the input text by the image generation process; The search unit, by the processor, Searching for the similar image from the image / sentence database using the generated image as a query to obtain sentences related to the similar image; The image generation assistant unit, by the processor, Extracting a plurality of keywords from the related sentences; 3. The image generating and searching device according to claim 2, wherein the keyword selected by a user from the plurality of keywords is added to the input sentence, and the generated image is obtained again by the image generating process.

10. The image generation unit, by the processor, Obtaining the generated image from the initial sentence; The image generation assistant unit, by the processor, Obtaining a modified sentence input by a user, and extracting existing elements and newly added additional elements contained in the initial sentence; A detection area obtained by detecting the object of the existing element from the generated image is reflected in a mask image as a retention area; The search unit, by the processor, searching the image / sentence database for the similar image using the corrected sentence as a query; The image generation assistant unit, by the processor, Detecting objects of the existing element and the additional element from the similar image, respectively; updating the mask image according to the frequency of appearance of the additional element; Calculating the relative size of the additional element to the existing element; The image generating and searching device according to claim 2 , wherein the mask image is generated according to the relative size.

11. The image and text database includes: The image generating and searching device according to claim 10, characterized in that the generated mask image is stored.

12. The image generating and searching device according to claim 1 ; an input device for specifying search conditions; a display device for displaying the search results; 1. An image generation and retrieval system comprising:

13. The display device includes: an operation screen for performing the image generation process and the search; The operation screen includes: The image generation and retrieval system according to claim 12, further comprising a generation process visualization field for displaying the image generation process visualized by the display unit.

14. The operation screen includes: a text input field for inputting the input text via the input device; an image generation button for acquiring the generated image via the input device; a generated image display field for displaying the generated image; a search condition field for specifying the search condition via the input device; an image search button for performing the search via the input device; an image search result field for displaying the search results; The image generation and retrieval system of claim 13, further comprising:

15. an image generation step of acquiring a new generated image from an input sentence and an input image by an image generation process by the processor; a sentence registration step of calculating a sentence feature from the input sentence by the processor; an image registration step of calculating image features from the input image and the generated image by the processor; a storing step of storing the input sentence, the input image, the generated image, the image feature amount, the sentence feature amount, and an input / output relationship in the image generation process in an image / sentence database as an image generation process; a search step of calculating a similarity between the text feature amount and the image feature amount by the processor, and searching the image / text database for an image similar to the search target using the similarity; a display step of visualizing the image generation process stored in the image and text database by the processor; 13. An image generation and retrieval method comprising:

Images