Image processing device, image processing method, program, and system

Abstract

To provide an image processing device capable of generating an image that is close to an image intended by a prompt providing side when generating an image from a prompt.SOLUTION: An image processing device according to the present disclosure includes acquisition means for acquiring, from a predetermined external device, information on the degree of interpretation by the predetermined external device of a prompt including an explanation about an image to be generated by a generative model, modification means for modifying the prompt based on the information on the degree of interpretation, and transmission means for transmitting the modified prompt to the predetermined external device via communication means. The modification means modifies the explanation of the prompt so as to increase the explanation reflected in the image generated by the generative model.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and a system. [Background technology]

[0002] In recent years, there has been remarkable progress in technology for generating images using models trained by machine learning. Technology for editing or generating images by inputting textual instructions for image generation called "prompts" into a machine learning model for image generation (also simply referred to as an image generation model) is becoming widespread. Patent Document 1 discloses a technology for changing the facial expression of a subject as desired by inputting an image and text related to emotions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-175923 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if the data used for learning or the degree of learning differs between image generation models, differences in the interpretation of the prompt may occur even when the same prompt is input to a model with the same structure. Therefore, when the same prompt is used to generate images using the image generation models of multiple devices, the prompt may be interpreted differently on each device, and some devices may generate images that differ from the intention of the prompt provider.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that, when generating an image from a prompt, can generate an image that is close to the image intended by the person providing the prompt. [Means for solving the problem]

[0006] In order to solve this problem, for example, an image processing device of the present invention has the following configuration: namely, it has an acquisition means that acquires, from a predetermined external device, information about the degree of interpretation by the predetermined external device of a prompt that includes an explanation about an image to be generated by the generative model, a correction means that corrects the prompt based on the information about the degree of interpretation, and a transmission means that transmits the corrected prompt to the predetermined external device via a communication means, wherein the correction means corrects the explanation of the prompt so as to increase the amount of the explanation reflected in the image generated by the generative model. [Effects of the Invention]

[0007] According to the present invention, when an image is generated from a prompt, it is possible to generate an image that is closer to the image intended by the prompt provider. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration according to each embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of a prompt communication device according to a first embodiment. [Figure 3] 10 is a flowchart showing the operation of image generation processing in the prompt communication device of the first embodiment. [Figure 4] 1 is a flowchart showing the operation of the prompt interpretation degree calculation process according to the first embodiment; [Figure 5] FIG. 10 is a diagram illustrating an example of image generation by the prompt communication device of the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of an image displayed by the prompt communication device of the first embodiment. [Figure 7] 10 is a flowchart showing the operation when a weight value is used in the prompt interpretation degree calculation process of the first embodiment. [Figure 8] FIG. 10 is a block diagram showing an example of the functional configuration of a prompt communication device according to a second embodiment. [Figure 9]10 is a flowchart showing the operation of an image generation process in a prompt communication device according to a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of image generation by the prompt communication device of the second embodiment. [Figure 11] FIG. 10 is a diagram showing an example of the functional configuration of a prompt communication system according to a third embodiment. [Figure 12A] 10 is a flowchart showing the operation of the initial processing in the prompt communication device 100A of the third embodiment. [Figure 12B] 10 is a flowchart showing the steady-state operation of the prompt communication device 100B of the third embodiment. [Figure 12C] 10 is a flowchart showing the operation of a routine process in the prompt communication device 100A of the third embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of image generation in the prompt communication system of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment 1) Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the following, an example of an image processing device in each embodiment will be described, in which a communication device (also referred to as a prompt communication device) capable of handling prompts for generating an image (also referred to simply as image generation prompts; details will be described later) is used. This embodiment can be applied to other devices capable of handling image generation prompts. These devices may include, for example, digital cameras, mobile phones including smartphones, game consoles, personal computers, tablet terminals, and other wearable information terminals.

[0011] <Overview of using prompts for generating images> Many images are taken with devices such as digital cameras and smartphones. However, capturing images often requires high photographic skill to capture the photographer's experiences and emotions, and it is impossible to capture events that did not exist at the time of the photograph. As a result, photographic images alone may not adequately convey the photographer's intended message. Instead of taking images, technologies are now available that allow users to generate desired images by inputting image generation prompts into an image generation model. In particular, multimodal AI technology, which can simultaneously process multiple different types of information, such as text, images, audio, and video, is advancing. Multimodal AI technology can now generate images that reproduce the content of text and generate text that describes the scene from the image content with high accuracy using a single generative model, enabling bidirectional conversion between images and text.

[0012] An image generation prompt is data that can be input to an image generation model and includes at least text information consisting of a sentence or a list of words. An example of an image generation prompt that can be written is, "The image shows a dog running on a lawn. The dog is an Akita with brown fur." ​​(See below). That is, the image generation prompt includes a description of the image to be generated by the image generation model, written in natural language. The description of the image in the image generation prompt may include, for example, a word that represents the object (subject) included in the image to be generated, or a description of the object. The image generation prompt can be used as an instruction for generating an image to the image generation model.

[0013] Because prompts are written in natural language, they can easily describe emotions and atmospheres that are difficult to express in images obtained by photography. Furthermore, by using a generative model, high-resolution images can be generated that are faithful to the information contained in the prompt. In other words, by sharing prompts with others and using images generated from the prompts, it becomes easier to convey experiences and emotions to others. On the other hand, as described above, when the same prompt is used to generate images using the image generation models of multiple devices, the prompts may be interpreted differently on each device. Furthermore, some devices may generate images that differ from the intent of the prompt provider. Therefore, the following embodiments specifically describe a technique that allows a user to easily grasp the degree to which a device has interpreted a prompt, and a technique that, when generating an image from a prompt, generates an image that closely resembles the intent of the prompt provider.

[0014] <Hardware Configuration Example of This Embodiment> 1 shows an example of the hardware configuration of a prompt communication device in this embodiment. The prompt communication device 100 includes, for example, a system bus 101, a CPU 102, a ROM 103, a RAM 104, an HDD 105, a GPU 106, an input device 107, a communication device 108, and a display device 109. Each unit of the prompt communication device 100 is connected to the system bus 101.

[0015] The CPU 102 includes one or more processors, such as a CPU (Central Processing Unit), and controls each function of the prompt communication device 100 by loading computer programs stored in the ROM 103 or HDD 105 into the RAM 104 and executing them. The ROM 103 includes a non-volatile storage medium, such as a semiconductor memory, and stores, for example, programs executed by the CPU 102 and necessary data. The RAM 104 includes a volatile storage medium, such as a semiconductor memory, and temporarily stores, for example, calculation results of the CPU 102. The HDD 105 includes a hard disk drive and stores, for example, computer programs executed by the CPU 102 and the processing results thereof. In this example, the prompt communication device 100 is described as having a hard disk, but the prompt communication device 100 may also include a storage medium such as an SSD instead of a hard disk.

[0016] The GPU (Graphics Processing Unit) 106 includes one or more processors and can execute, for example, part or all of the processes in the learning stage and inference stage of an image generation model. Compared to a CPU, a GPU can process more data in parallel. The learning stage and inference stage of an image generation model may involve repeated calculations using a large-scale neural network containing a large number of neurons, so it is effective to use a GPU for the processing.

[0017] The input device 107 includes operation members such as buttons and a touch panel that accept operation inputs to the prompt communication device 100. The communication device 108 communicates with external devices via a network such as the Internet. The display device 109 includes a display panel such as an OLED, and displays, for example, images and text information.

[0018] <Example of functional configuration of prompt communication device> Fig. 2 shows an example of the functional configuration of the prompt communication device 100 in this embodiment. Note that the example functional configuration shown in Fig. 2 focuses on the configuration for generating an image using an image generation prompt and displaying the generated image. Therefore, the configuration of the prompt communication device 100 is not limited to the configuration shown in Fig. 2. Some or all of the example functional configuration shown in Fig. 2 may be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program.

[0019] The prompt communication device 100 includes, for example, a prompt acquisition unit 201, a prompt interpretation degree calculation unit 202, a prompt search unit 203, a prompt database 204, a prompt / image conversion unit 205, and an image data display unit 206.

[0020] The prompt acquisition unit 201 acquires an image generation prompt from an external device via the communication device 108 and outputs it to the RAM 104. In the present embodiment, an example will be described in which an image generation prompt generated by an external device is acquired from the external device via the communication device 108. However, the configuration of the prompt acquisition unit 201 is not limited to this, and for example, the prompt for image generation recorded on an external server or cloud via communication with an external network may be acquired. Alternatively, the prompt for image generation recorded in the external storage device may be acquired by accessing an external storage device such as a USB memory or an external HDD.

[0021] The prompt interpretation degree calculation unit 202 calculates the degree of interpretation of the input image generation prompt. The interpretation degree of the image generation prompt (also simply referred to as the prompt interpretation degree) and the calculation method will be described later. The prompt interpretation degree calculation unit 202 is composed of, for example, a prompt search unit 203 and a prompt database 204. The prompt search unit 203 searches the prompt database 204 for the content described in the input image generation prompt. The prompt database 204 is stored in the HDD 105 and contains expressions that can be reflected in images by a generation model. In other words, the prompt database 204 is a group of information on prompts that can be converted into image data by the prompt / image conversion unit 205, which will be described later. The prompt interpretation degree calculation unit 202 calculates and outputs the prompt interpretation degree based on the search results of the prompt database 204.

[0022] The prompt / image conversion unit 205 reads an image generation prompt from the RAM 104 and converts the image generation prompt into image data (i.e., generates image data corresponding to the image generation prompt). The prompt / image conversion unit 205 outputs the obtained image data to the RAM 104. The conversion into image data is realized, for example, by the GPU 106 reading a prompt / image conversion model from the ROM 103 and performing arithmetic processing. The prompt / image conversion model is, for example, a machine learning model that has learned the relationship between various image generation prompts and images (i.e., has learned to generate images corresponding to various image generation prompts from the prompts). Note that known techniques related to image generation models can be applied to the processing in the learning and inference stages of the prompt / image conversion model, and therefore a description thereof will be omitted in this embodiment. Furthermore, this embodiment will be described taking as an example a case where the processing of the prompt / image conversion unit 205 is executed in the prompt communication device 100. However, the function of the prompt / image conversion unit 205 may be provided externally, and the processing of the prompt / image conversion unit 205 may be realized, for example, on an external server or cloud.

[0023] The image data display unit 206 displays various image data and text information on the display device 109. In this embodiment, at least the image data converted by the prompt / image conversion unit 205 is displayed.

[0024] <Image generation process using image generation prompts> The operation of the image generation process using the image generation prompt in the prompt communication device 100 of this embodiment will be described with reference to FIGS.

[0025] Fig. 3 shows the overall operation of the image generation process using the image generation prompt, and Fig. 4 shows the operation of the prompt interpretation degree calculation process of this embodiment. This process can be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program to cause the various units shown in Fig. 2 to function.

[0026] In S301, the prompt acquisition unit 201 acquires an image generation prompt from an external device via the communication device 108 and outputs it to the RAM 104. FIG. 5(A) shows an example of the acquired image generation prompt. The image generation prompt shown in FIG. 5(A) includes information such as "The image shows a dog running on the lawn. The dog is an Akita dog with brown fur," as an explanation of the image to be generated by the image generation model.

[0027] In S302, the prompt interpretation degree calculation unit 202 reads the image generation prompt acquired in S301 from the RAM 104, calculates the prompt interpretation degree, and outputs it to the RAM 104. Now, the operation of the prompt interpretation degree calculation process will be described with reference to FIG.

[0028] <Prompt Interpretation Calculation Process Operation> Similar to the image generation process, the prompt interpretation degree calculation process can be realized by, for example, the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program to cause the various units shown in FIG. 2 to function. The prompt interpretation degree calculation process is started when the above-mentioned step S302 is executed. Note that the following explanation uses an example in which a prompt interpretation degree is calculated, but as long as the prompt interpretation degree can be determined, the value may be obtained using a table, not just in the case of calculating the prompt interpretation degree.

[0029] In S401, the prompt search unit 203 divides the input image generation prompt into units for calculating the degree of interpretation. The prompt interpretation degree may be calculated for each word that constitutes the image generation prompt (i.e., the explanation of the image), or for each phrase, clause, or sentence. FIG. 5B shows the divided image generation prompt and the modifying relationships between the words. For example, the prompt search unit 203 divides the image generation prompt shown in FIG. 5A into units of words such as "grass," "run," "one," "dog," "brown," and "Akita dog" as shown in FIG. 5B.

[0030] In S402, the prompt search unit 203 uses the prompt database 204 to search the input image generation prompt in the units divided in S401. The prompt database 204 stores a group of prompt (text) information that the prompt / image conversion unit 205 can convert into image data. The prompt / image conversion unit 205 cannot convert text information that does not exist in the prompt database 204 into image data. For this reason, text information that does not exist in the prompt database 204 is content that the prompt communication device 100 cannot interpret (cannot reflect in an image). Therefore, the prompt communication device 100 can use the prompt database 204 to search for units into which the image generation prompt has been divided (i.e., parts of the explanation for the image), thereby determining whether the units can be reflected in the image.

[0031] In this embodiment, we will explain an example in which, among the contents divided in S402, "Akita dog" does not exist in the prompt database 204, and therefore the target text cannot be found when the prompt search unit 203 searches for "Akita dog."

[0032] In S403, the prompt search unit 203 calculates the degree of interpretation of the image generation prompt. For example, the prompt search unit 203 calculates the degree of prompt interpretation for each prompt divided in S401 based on the prompt search results acquired in S402. The prompt search unit 203 outputs the calculated prompt interpretation degree to RAM 104. After outputting the prompt interpretation degree to RAM 104, the prompt search unit 203 ends this process and returns to the caller.

[0033] FIG. 5(B) shows the modifying relationships for the image generation prompts divided into words in S401. For example, information such as the breed of dog "Akita" and the coat color "brown" is qualitative information that indicates the appearance, attributes, and characteristics of the "dog." Information indicating movement and information about the number of "dogs," such as "one," is instantaneous information that describes the state of the "dog" over a specific period of time. All of this information is written to modify the "dog."

[0034] In this embodiment, the prompt interpretation degree represents, for example, whether a specific word or phrase can be interpreted, or the ratio of whether a specific word or phrase can be interpreted as information modifying the word or phrase. For example, the prompt interpretation degree for "Akita dog" is 0% (or a value close to 0%). As shown in FIG. 5B, the image generation prompt in FIG. 5A describes four pieces of information modifying the word "dog": breed, coat color, movement, and number. If the breed "Akita dog" is determined to be ininterpretable, while the coat color "brown," the movement "running," and the number "one" are determined to be interpretable, the interpretation degree for the word "dog" is 3 / 4. Therefore, the prompt search unit 203 can calculate the interpretation degree for the word "dog" as 75%. In this way, the prompt interpretation degree represents, for example, the degree to which the explanation about the image is reflected in the image.

[0035] In this embodiment, an example of a method for calculating the prompt interpretation degree for a specific word or phrase has been described. For a clause or sentence, the interpretation degree may be defined as the ratio of the possible interpretations of the words or phrases that make up the clause or sentence. The above description also deals with an example in which all information is treated as a single unit. However, when calculating the prompt interpretation degree, weight values ​​set for each word, phrase, clause, or sentence in the image generation prompt may also be taken into account. Calculation of the prompt interpretation degree using weight values ​​will be described later.

[0036] 3 again, in S303, the prompt / image conversion unit 205 reads out the image generation prompt acquired in S301 from the RAM 104, converts it into image data, and outputs it to the RAM 104.

[0037] If the image generation prompt in FIG. 5(A) is sufficiently interpreted (e.g., if the prompt interpretability is sufficiently high), the process in S303 is expected to generate image data such as that shown in FIG. 5(C). If the prompt contains a word or phrase that cannot be interpreted, an image can be generated by, for example, ignoring the content or substituting an alternative word that fits the context. In this embodiment, for example, because "Akita dog" cannot be interpreted, the prompt / image conversion unit 205 generates image data by substituting "Siberian Husky," which is a different breed of "Akita dog" (existing in the prompt database 204). For example, the prompt / image conversion unit 205 generates image data such as that shown in FIG. 5(D). Note that when substituting words, the prompt / image conversion unit 205 may acquire information on the type of modifier, such as breed, coat color, movement, or number, for each word. For example, the prompt / image conversion unit 205 may estimate the type of modifier of a word using a learning model. When substituting the uninterpretable word "Akita dog", the prompt / image conversion unit 205 may use the estimated type of modifier to search a database for words of the same type of modifier (e.g., dog breed) as the uninterpretable word, and substitute them with the interpretable word "Siberian Husky".

[0038] In S304, the image data display unit 206 reads the image generation prompt, information about the prompt interpretation level, and image data obtained in S301 to S303 from RAM 104 and displays them on the display device 109. The information about the interpretation level may be, for example, information indicating an unknown word in the prompt, such as "Akita dog." In other words, the information about the interpretation level may include, for example, a part of the description of the image in the prompt (for example, "Akita dog") that does not meet the criteria for the degree of reflection in the image.

[0039] FIG. 6 shows an example of the image display displayed in S304. For example, the image data display unit 206 displays the image generation prompt in text format in an area 601. The image data display unit 206 also displays information about the prompt interpretation level in an area 602, separate from the area 601. For example, if the prompt contains at least one of a word, phrase, clause, or sentence whose prompt interpretation level is equal to or lower than a predetermined threshold, the image data display unit 206 notifies the user by displaying the word, phrase, clause, or sentence in the area 602. At this time, the image data display unit 206 can highlight the word, phrase, clause, or sentence displayed in the area 602. The predetermined threshold is set for the purpose of notifying the user when the image data converted in S303 contains a subject whose reproducibility with respect to the input image generation prompt is low. In this embodiment, as an example, the predetermined threshold is set so that the user is notified of content (e.g., a word) whose prompt interpretation level is 0%. As a result, the image data display unit 206 displays "Akita dog" in the area 602. Furthermore, the image data display unit 206 may display the image data converted in S303 in the area 603. The image data displayed in the area 603 is image data converted only with the prompt explanation that the generative model can reflect in the image. When the image data display unit 206 displays the image data etc. on the display device 109, it ends the series of operations related to the image generation process.

[0040] In this way, the image generation process according to this embodiment calculates the degree of interpretation of the acquired image generation prompt and makes it possible to notify the user according to the calculation result. Note that in this embodiment, an example has been described in which a notification according to the degree of interpretation of the image generation prompt is displayed on the display device 109. However, the notification according to the degree of interpretation is not limited to the above example, and for example, a notification may be sent to an external device via the communication device 108.

[0041] <Operation when weight values ​​are used in the prompt interpretation degree calculation process> As described above, the prompt interpretation degree calculation unit 202 may take into account the weight value of the prompt for image generation when calculating the prompt interpretation degree. In this case, the weight value may be set in units of words, phrases, clauses, or sentences (i.e., parts of the explanation for the image) included in the prompt for image generation. The prompt interpretation degree calculation process using a weight value will be described with reference to FIG. 7. Like the prompt interpretation degree calculation process shown in FIG. 4, the prompt interpretation degree calculation process shown in FIG. 7 can be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program to cause the various units shown in FIG. 2 to function. Note that processes similar to those shown in FIG. 4 are assigned the same reference numerals, and their description will be omitted.

[0042] In S401, the prompt search unit 203 divides the input image generation prompt into units for calculating the degree of interpretation, as described above. In S701, the prompt search unit 203 determines whether a weight value is set for each division unit of the image generation prompt (i.e., a part of the explanation for the image). For example, if a weight value is associated with each division unit of the image generation prompt, the prompt search unit 203 can determine that a weight value has been set. If the prompt search unit 203 determines that a weight value has been set, the process proceeds to S703; otherwise, the process proceeds to S702.

[0043] In S702, the prompt search unit 203 sets a weight value for each division unit of the image generation prompt. The weight value of the prompt may be determined, for example, based on the importance of the target word, or clause, phrase, or sentence. For example, the prompt search unit 203 sets a high weight value for a word that is the subject of the prompt, a word that may be the main subject of the image data generated by the prompt / image conversion unit 205, or a clause or sentence that includes such a word. For example, the prompt search unit 203 sets a higher weight value for the subject words "grass" and "dog" in the image generation prompt of FIG. 5(A) than for other information. Alternatively, the prompt search unit 203 may set a high weight value for clauses or sentences that include "grass" or "dog." Alternatively, the prompt search unit 203 may set a high weight value for a unit (e.g., a word) that contains a lot of information modifying a word that represents a noun in the image generation prompt. For example, in the image generation prompt of FIG. 5A, there is no information modifying "grass," but information such as the breed, coat color, and behavior of the dog is described for "dog." In this case, the prompt search unit 203 sets a higher weight value for "dog" than for "grass." That is, when weighting the image generation prompt shown in FIG. 5A, the prompt search unit 203 sets the highest weight value for "dog," and the next highest weight value for "grass." The prompt search unit 203 then sets the lowest weight value for information that modifies "dog," such as "one," "brown," and "Akita dog." Note that known natural language analysis methods and trained classification models can be used to estimate which of the division units (e.g., words) of the image generation prompt are the subject, which can be the main subject, and which are modifying information.

[0044] In addition, in this embodiment, an example has been described in which some of the prompts for image generation correspond to being reflected in the generated image (hereinafter also referred to as positive prompts). However, there are also cases in which some of the prompts for image generation correspond to not being reflected in the generated image (hereinafter referred to as negative prompts). In this case, the prompt search unit 203 may set a weight value for words, phrases, clauses, or sentences belonging to negative prompts higher than the weight value set for positive prompts, so that the degree of interpretation of negative prompts is calculated with priority.

[0045] Furthermore, the prompt search unit 203 may assign different weight values ​​to the division units of the image generation prompt depending on specific syntax or emphasis expressions. For example, the prompt search unit 203 may assign a higher weight value to an earlier description order in the image generation prompt, or may assign a higher weight value according to the order of word appearance frequency. The prompt search unit 203 may assign a relatively high weight value to expressions that use exclamation marks or expressions that are emphasized by rhetoric.

[0046] In S703, the prompt search unit 203 searches the database 204 for division units of the image generation prompt in descending order of weight value. The search of the database 204 can be performed in the same manner as the process described above in S402. Note that the prompt search unit 203 may limit the prompts to be searched to those with a set weight value equal to or greater than a predetermined threshold. In this way, the number of searches can be limited, thereby speeding up the process.

[0047] In S704, the prompt search unit 203 calculates the degree of interpretation for each divided image generation prompt based on the prompt search results acquired in S703. In S704, the prompt search unit 203 calculates the degree of interpretation by assigning a weight to the ratio of the likelihood of interpretation to information modifying a specific word or phrase. For example, in the example shown in FIG. 5B, the image generation prompt of FIG. 5A describes four pieces of information modifying the word "dog." Furthermore, when the image generation prompt of FIG. 5A is acquired from an external device, the external device has already set a weight value for information about the dog breed at twice the weight of the other information (brown, running, one). If the dog breed "Akita" cannot be interpreted, the degree of interpretation for the word "dog" is 3 / 5. Therefore, the prompt search unit 203 calculates the degree of interpretation for the word "dog" to be 60%.

[0048] In this way, in the prompt interpretation degree calculation process, by using a weight value to calculate the interpretation degree of the acquired image generation prompt, it is possible to prioritize the interpretation degree for important information in the image generation prompt.

[0049] As described above, in this embodiment, the prompt communication device 100 determines the degree of interpretation of the acquired image generation prompt (i.e., explanation of the image). Then, based on the degree of interpretation of the prompt, the prompt communication device 100 displays on the display device 109 information about the degree of interpretation of the prompt (e.g., "Akita dog") whose degree of reflection in the image does not satisfy a predetermined standard. In this way, the user can understand the degree of interpretation of the prompt by the prompt device 100.

[0050] <Embodiment 2> Next, a second embodiment of the present invention will be described. This embodiment corresponds to a case where the prompt communication device 100 described in the first embodiment does not have the prompt database 204. An example of a case where the prompt database 204 is not included is when the internal processing of the image / prompt conversion model used in the prompt / image conversion process is kept confidential, making it impossible to grasp the internal processing. Alternatively, this also applies to a case where the prompt communication device 100 does not have the prompt / image conversion unit 205 and the prompt / image conversion process is performed on an external server or cloud. Note that in this embodiment, the configuration of the prompt interpretation degree calculation unit and the processing performed by this unit differ from those in the first embodiment, but the other configurations are similar to those in the first embodiment. Therefore, components that are the same or substantially the same as those in the first embodiment are given the same reference numerals, and their description will be omitted.

[0051] Fig. 8 shows an example of the functional configuration of the prompt communication device 100 that calculates the degree of interpretation of a prompt in this embodiment. Furthermore, part or all of the example functional configuration shown in Fig. 8 may be realized by, for example, the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program.

[0052] The prompt communication device 100 includes, for example, a prompt acquisition unit 201, a prompt interpretation degree calculation unit 801, and an image data display unit 206.

[0053] The prompt interpretation degree calculation unit 801 calculates the degree of interpretation of the input image generation prompt. In this embodiment, the prompt interpretation degree calculation unit 801 includes a prompt / image conversion unit 205, an image / prompt conversion unit 802, and a prompt comparison unit 803. The prompt / image conversion unit 205 is realized with the same configuration as in the first embodiment.

[0054] The image / prompt conversion unit 802 reads the image / prompt conversion model from the RAM 104 and generates a prompt for generating an image from the image data. The calculation processing of the image / prompt conversion unit 802 is performed, for example, by the GPU 106. The image / prompt conversion model is, for example, a machine learning model that has learned the relationship between various images and prompts for generating images (i.e., has learned to generate prompts corresponding to various images from the images). The conversion model used by the prompt / image conversion unit 205 and the image / prompt conversion unit 802 is a single generation model that enables bidirectional conversion between image data and prompts.

[0055] The prompt comparison unit 803 receives at least two image generation prompts and compares the contents of the prompts. Through the comparison, the prompt comparison unit 803 can identify words and the like that do not exist in one of the image generation prompts.

[0056] <Image generation process using image generation prompts> Next, the image generation process of this embodiment will be described with reference to Fig. 9. The image generation process of this embodiment can be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program to cause each unit of the prompt communication device 100 to function.

[0057] In S301, the prompt acquisition unit 201 acquires a prompt for image generation and outputs it to the RAM 104, as in the first embodiment. At this time, it is assumed that the prompt A for image generation shown in FIG. 10A is acquired. Also, in S303, the prompt / image conversion unit 205 reads the prompt A for image generation from the RAM 104, converts it into image data, and outputs it to the RAM 104, as in the first embodiment. In this embodiment, the prompt / image conversion unit 205 converts the image generation prompt A into image data without being able to interpret the dog breed "Akita dog." As a result, it is assumed that image data such as that shown in FIG. 10B is generated. For example, image data is generated in which "Akita dog" is replaced with another dog breed, "Siberian Husky."

[0058] In S901, the image / prompt conversion unit 802 reads the image data converted in S303 from the RAM 104, converts it into image generation prompt B, and outputs it to the RAM 104. For example, the image / prompt conversion unit 802 converts image data (FIG. 10(B)) that was converted in a state where the dog breed "Akita" could not be interpreted into a prompt for image generation. As a result, it is assumed that image generation prompt B such as that shown in FIG. 10(C) is generated. In other words, an image generation prompt is generated that does not include information about the dog breed "Akita."

[0059] In S902, the prompt comparison unit 803 reads out the image generation prompt A acquired in S301 and the image generation prompt B converted in S901 from the RAM 104. The prompt comparison unit 803 then compares the two read image generation prompts to calculate the degree of interpretation of the image generation prompt A.

[0060] When comparing image generation prompt A and image generation prompt B, the dog breed "Akita" is only included in image generation prompt A. Therefore, if image generation prompt B does not contain information identical to the words, phrases, clauses, or sentences that make up image generation prompt A, prompt comparison unit 803 can determine that image generation prompt A is a prompt that contains information that cannot be interpreted.

[0061] In this way, by comparing the content of the input image generation prompt with the content converted into image data and then converted back into a prompt, it is possible to determine whether the image generation prompt can be interpreted. That is, the prompt comparison unit 803 can determine that the dog breed "Akita" is ininterpretable, but the coat color "brown," the movement "one," and the number "one" are interpretable. After identifying the ininterpretable content, the degree of interpretation can be calculated based on whether the content can be interpreted, as in the first embodiment.

[0062] In S304, the image data display unit 206 reads the image generation prompt A, the prompt interpretation degree, and the image data from the RAM 104, as in the first embodiment, and displays them on the display device 109 according to the calculation result of the prompt interpretation degree.

[0063] In this way, in the image generation process according to this embodiment, the acquired prompt is converted into an image (the image is generated), the generated image is converted into a prompt to reproduce the prompt, and the acquired prompt and the reproduced prompt are compared. Then, based on the comparison result of the prompts, the degree of interpretation of the acquired prompt is calculated. This also makes it possible to calculate the degree of interpretation of the prompt and notify the user according to the calculation result. In other words, the user can grasp the degree of interpretation of the prompt by the prompt device 100.

[0064] <Embodiment 3> Next, a third embodiment will be described. Fig. 11 shows an example of the functional configuration of a prompt communication device 100A and a prompt communication device 100B as an example of a prompt communication system for transmitting and receiving prompts between devices in this embodiment. The hardware configuration of the prompt communication device 100A and the prompt communication device 100B is the same as that of the prompt communication device 100 described above with reference to Fig. 1. For convenience of distinguishing between the prompt communication devices, the prompt communication devices are given reference numerals with an A or B added. Note that part or all of the example functional configuration shown in Fig. 11 may be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100A or the prompt communication device 100B executing a computer program.

[0065] The prompt communication device 100A includes, for example, a prompt acquisition unit 1101A, a prompt correction unit 1102A, a communication unit 1103A, and a prompt / image conversion unit 1104A. The prompt communication device 100B includes, for example, a prompt interpretation degree determination unit 1101B, a communication unit 1102B, a prompt / image conversion unit 1103B, and an image data display unit 1104B.

[0066] The prompt acquisition unit 1101A acquires an image generation prompt from an external device via the communication device 108 and outputs it to the RAM 104 of its own device. In the present embodiment, an example will be described in which the image generation prompt is acquired from the external device via the communication device 108. However, the configuration of the prompt acquisition unit 1101A is not limited to this, and for example, the prompt acquisition unit 1101A may be provided with a communication unit for connecting to an external network and acquire an image generation prompt recorded on an external server or cloud. Alternatively, the prompt acquisition unit 1101A may have a connection unit for connecting to an external storage device and acquire an image generation prompt recorded in an external storage device such as a USB memory or an external HDD.

[0067] The prompt correction unit 1102A reads the image generation prompt and the prompt interpretation level for the image generation prompt from the RAM 104 of its own device, and performs a predetermined correction process on the image generation prompt based on the prompt interpretation level. The prompt correction unit 1102A outputs the image generation prompt that is the result of the correction process to the RAM 104 of its own device. Details of the correction process will be described later in the explanation of the prompt communication process.

[0068] The communication unit 1103A transmits and receives data to and from the prompt communication device 100B. For example, the communication unit 1103A reads data such as an image generation prompt from the RAM 104 of its own device and transmits it to the prompt communication device 100B, or receives data from the prompt communication device 100B and outputs it to the RAM 104 of its own device.

[0069] The communication unit 1102B transmits and receives data to and from the prompt communication device 100A. For example, the communication unit 1102B receives data from the prompt communication device 100A and outputs it to the RAM 104 of its own device, or reads data from the RAM 104 of its own device and transmits it to the prompt communication device 100A.

[0070] The prompt communication device 100A transmits image generation prompts output by the above-mentioned prompt acquisition unit 1101A and prompt modification unit 1102A to the prompt communication device 100B via the communication unit 1103A. The prompt communication device 100A may also transmit image data output by a prompt / image conversion unit 1104A (described below) to the prompt communication device 100B via the communication unit 1103A. The prompt communication device 100A also receives a prompt interpretation degree output by a prompt interpretation degree determination unit 1101B (described below) from the prompt communication device 100B via the communication unit 1103A.

[0071] Similarly, the prompt communication device 100B receives image generation prompts output by the prompt acquisition unit 1101A and the prompt modification unit 1102A from the prompt communication device 100A via the communication unit 1102B. The prompt communication device 100B may also receive image data output by the prompt / image conversion unit 1104A described above from the prompt communication device 100A. The prompt communication device 100B also transmits the prompt interpretation degree output by the prompt interpretation degree determination unit 1101B described above to the prompt communication device 100A via the communication unit 1102B.

[0072] Prompt / image conversion unit 1104A reads out a prompt for image generation from RAM 104 of its own device, converts it into image data (i.e., generates image data according to the prompt for image generation), and outputs it to RAM 104 of its own device. Similarly, prompt / image conversion unit 1103B reads out a prompt for image generation from RAM 104 of its own device, converts it into image data, and outputs it to RAM 104 of its own device.

[0073] Conversion into image data can be performed in the same manner as in the first embodiment, and is realized, for example, by the GPU 106 reading out the prompt / image conversion model from the ROM 103 of its own device and performing arithmetic processing.

[0074] The prompt interpretation degree determination unit 1101B reads the image generation prompt from the RAM 104 of its own device and calculates the prompt interpretation degree. Additionally, the prompt interpretation degree determination unit 1101B determines the prompt interpretation degree using the prompt interpretation degree determination criteria read from the RAM 104 of its own device. The prompt interpretation degree determination unit 1101B then outputs the determination result of the prompt interpretation degree to the RAM 104. Note that the calculation of the prompt interpretation degree is assumed to be performed using the method described above in either the first or second embodiment, and a description of this method will be omitted in this embodiment. The determination of the prompt interpretation degree will be described in detail in the prompt communication processing described below.

[0075] The image data display unit 1104B displays various image data on the display device 109. In this embodiment, the image data display unit 1104B displays, for example, image data converted by the prompt / image conversion unit 1104A and the prompt / image conversion unit 1103B.

[0076] <Prompt communication processing operation> The prompt communication process of the prompt communication device 100A and the prompt communication device 100B of this embodiment will be described with reference to FIGS. 12A to 12C.

[0077] 12A to 12C show the operation of the prompt communication process of this embodiment. The processes shown in FIGS. 12A and 12C can be realized by the units shown in FIG. 11 when the CPU 102 or GPU 106 of the prompt communication device 100A executes a computer program. The process shown in FIG. 12B can be realized by the units shown in FIG. 11 when the CPU 102 or GPU 106 of the prompt communication device 100B executes a computer program. The processes shown in FIGS. 12A to 12C, which can be executed sequentially from the process shown in FIG. 12A, can basically be executed when a communication instruction is received from the user, or at any time before or after that. However, even when no communication instruction is received from the user, they can also be executed at regular intervals, for example, if the prompt communication device 100A and the prompt communication device 100B are always running and ready for communication.

[0078] <Initial Processing Operation of the Prompt Communication Device 100A> 12A shows the initial processing operations of the prompt communication device 100A. The processing steps from S1201 to S1203 are performed at the beginning of each prompt-related communication between the prompt communication device 100A and the prompt communication device 100B. After that, the prompt communication device 100A performs the regular processing operations from S1213 to S1218, which will be described later.

[0079] In S1201, the prompt acquisition unit 1101A acquires an image generation prompt from an external device via the communication device 108 and outputs it to the RAM 104.

[0080] In S1202, the prompt modification unit 1102A reads the image generation prompt acquired in S1201 from the RAM 104. Then, a criterion for determining the prompt interpretation degree for the image generation prompt is set. The prompt modification unit 1102A modifies the image generation prompt, for example, to set the criterion for determining the prompt interpretation degree. The prompt modification unit 1102A outputs the modified image generation prompt to the RAM 104.

[0081] The prompt interpretation degree determination criterion is information accompanying the image generation prompt, and is a threshold value for the prompt interpretation degree calculated in step S1205, which will be described later. The prompt interpretation degree determination criterion can be set for each word, clause, phrase, or sentence that constitutes the image generation prompt.

[0082] The prompt interpretation degree determination criteria can be determined or changed based on the communication history with the other device in the prompt communication. For example, if the prompt modification unit 1102A has a history of communication with the other device in the prompt communication, the prompt interpretation degree for that word on the other device may satisfy the determination criteria. In this case, the prompt modification unit 1102A can either not set the prompt interpretation degree determination criteria or set the determination criteria to a lower value (than when there is no communication history).

[0083] On the other hand, the prompt correction unit 1102A may set a higher prompt interpretation degree criterion for words representing the subject, words representing the main subject, or clauses or sentences containing these words among the image generation prompts than for other parts. Furthermore, the prompt interpretation degree criterion may be set higher for words representing nouns that contain a predetermined number or more of information that embodies the word, such as modifiers, or for clauses or sentences containing such words than for other parts. In this case, the other device may be requested to output a high interpretation degree for the word. In other words, the other device is required to have a generation model with high reproducibility for the word, or to generate an image that accurately reproduces the word by repeatedly receiving corrected prompts.

[0084] In this embodiment, the prompt communication device 100A sets the prompt interpretation degree determination criteria. However, for example, the prompt interpretation degree determination criteria may be set in advance and stored in the RAM 104 of the prompt communication device 100A before the image generation prompt is acquired in S1201. Alternatively, the prompt communication device 100B may set the prompt interpretation degree determination criteria.

[0085] In S1203, the communication unit 1103A reads the prompt for image generation corrected in S1202 from the RAM 104 and transmits it to the prompt communication device 100B. Once the prompt for image generation has been transmitted, the prompt communication device 100A ends the initial processing.

[0086] <Operation of Steady-State Processing of Prompt Communication Device 100B> Next, the steady-state processing in the prompt communication device 100B will be described with reference to FIG. 12B.

[0087] In S1204, the communication unit 1102B receives the image generation prompt transmitted from the prompt communication device 100A in S1203 and S1216 (described later), and outputs it to its own device's RAM 104. In S1205, the prompt interpretation degree determination unit 1101B reads the image generation prompt received in S1204 from its own device's RAM 104, and calculates the prompt interpretation degree.

[0088] In S1206, the prompt interpretation degree determination unit 1101B determines the prompt interpretation degree. Specifically, the prompt interpretation degree determination unit 1101B reads the prompt interpretation degree determination criteria set in S1202 from the RAM 104 of its own device, and determines whether the prompt interpretation degree calculated in S1205 satisfies this criteria. The prompt interpretation degree determination unit 1101B then outputs the prompt interpretation degree determination result to the RAM 104 of its own device. If there is even one prompt that does not satisfy the criteria, the prompt interpretation degree determination unit 1101B determines that the prompt interpretation degree does not satisfy the criteria, and proceeds to S1207. On the other hand, if there is no prompt that does not satisfy the criteria, the prompt interpretation degree determination unit 1101B determines that the prompt interpretation degree satisfies the criteria, and proceeds to S1209.

[0089] In S1207, the prompt interpretation degree determination unit 1101B determines whether image data will be received from the prompt communication device 100A (i.e., whether a specific condition for receiving image data is met). For example, the prompt interpretation degree determination unit 1101B determines that image data will be received if the number of prompts that do not satisfy the determination criterion for the prompt interpretation degree determination result determined in S1206 exceeds a predetermined number. Alternatively, the prompt interpretation degree determination unit 1101B may determine that image data will be received if the number of times when at least one prompt does not satisfy the determination criterion exceeds a predetermined number when the operations of S1204 to S1209 are repeated. Alternatively, the prompt interpretation degree determination unit 1101B may determine that image data will be received if an instruction (e.g., an instruction to receive an image) is received from the prompt communication device 100A. Furthermore, the respective determination contents may be combined. If the prompt interpretation degree determination unit 1101B determines that image data will be received, the process proceeds to S1211; otherwise, the process proceeds to S1208.

[0090] In S1208, the communication unit 1102B reads out the determination result of the prompt interpretation degree determined in S1206 from the RAM 104 of its own device, and transmits the determination result of the prompt interpretation degree to the prompt communication device 100A.

[0091] Steps S1209 to S1210 are the processing steps until the image data is displayed on the display device. In this embodiment, even if the result of the prompt interpretation degree determination in step S1206 indicates that some of the prompts do not satisfy the determination criteria, the image generation prompt is displayed. However, the image generation prompt may be finally displayed when there are no more prompts that do not satisfy the determination criteria.

[0092] In S1209, the prompt / image conversion unit 1103B reads the image generation prompt received in S1204 from the RAM 104 of its own device, converts it into image data, and outputs it to the RAM 104 of its own device. In S1210, the image data display unit 1104B reads the image data converted in S1209 from the RAM 104 of its own device, and displays it on the display device 109. Once the prompt communication device 100B displays the image data, it ends one round of regular processing.

[0093] In S1211, the communication unit 1102B transmits a request for image data to the prompt communication device 100A. In S1212, the communication unit 1102B receives the image data transmitted from the prompt communication device 100A in S1218 (described later) and outputs it to the RAM 104 of its own device.

[0094] <Operation of Steady-State Processing of Prompt Communication Device 100A> Next, the operation of the steady-state processing of the prompt communication device 100A will be described with reference to FIG. 12C.

[0095] In S1213, the prompt correction unit 1102A determines whether image data should be sent to the prompt communication device 100B (i.e., whether a specific condition for sending image data is met). For example, the prompt correction unit 1102A determines to send image data when it receives a request to send image data from the prompt communication device 100B. Alternatively, the prompt correction unit 1102A may determine to send image data when the number of prompts that do not satisfy the determination criterion for the determination result of the prompt interpretation level received in S1214 (described later) exceeds a predetermined number. Alternatively, the prompt correction unit 1102A may determine to send image data when the number of times when at least one prompt does not satisfy the determination criterion exceeds a predetermined number when the operations of S1213 to S1216 are repeated. Furthermore, each determination content may be combined. If the prompt correction unit 1102A determines to send image data, it proceeds to S1217; if it determines not to send image data, it proceeds to S1214.

[0096] In S1214, the communication unit 1103A receives the determination result of the prompt interpretation level sent from the prompt communication device 100B in S1208, and outputs it to the RAM 104 of its own device.

[0097] In S1215, the prompt correction unit 1102A reads the image generation prompt and the determination result of the prompt interpretation degree from the RAM 104 of its own device. Then, the prompt correction unit 1102A uses the determination result of the prompt interpretation degree to correct the image generation prompt so that the prompt interpretation degree is higher than the current prompt interpretation degree. In other words, the prompt correction unit 1102A corrects the image generation prompt so that the explanation about the image that is reflected in the generated image is increased. The prompt correction unit 1102A outputs the corrected image generation prompt to the RAM 104.

[0098] The image generation prompt can be modified, for example, by replacing the image generation prompt whose prompt interpretation degree does not satisfy the determination criterion in S1206 with an alternative word. That is, the prompt modification unit 1102A can replace a part of the image generation prompt (part of the description of the image) that does not satisfy the criterion with another description. Alternatively, the image generation prompt can be modified, for example, by adding a description to the image generation prompt whose prompt interpretation degree does not satisfy the determination criterion in S1206 (i.e., adding a description of the image). For example, a case will be described in which, for the image generation prompt shown in FIG. 13(A), it is determined in S1206 that "Akita dog" does not satisfy the determination criterion. In this case, the prompt modification unit 1102A modifies the image generation prompt by further adding a description about "Akita dog," as shown in FIG. 13(D). The additional information may include, for example, "It is a medium-sized dog," "Its ears are triangular with drooping tips," and so on, and may include descriptions that further explain the external characteristics of the "Akita dog" that was determined not to meet the criteria.

[0099] In S1216, the communication unit 1103A reads the image generation prompt corrected in S1215 from RAM 104 and transmits it to the prompt communication device 100B. In this way, the prompt communication device 100B can generate an image from the corrected image generation prompt, and can generate an image with a higher degree of interpretation (i.e., an image that is closer to the image intended by the communication device 100A).

[0100] In S1217, the prompt / image conversion unit 1104A reads the prompt for image generation from the RAM 104 of its own device, converts it into image data, and outputs it to the RAM 104 of its own device (to send an image to the prompt communication device 100B). Note that the conversion from the prompt for image generation to image data can be performed in the same manner as in embodiment 1. In S1218, the communication unit 1103A reads the image data converted in S1217 from the RAM 104 of its own device, and sends it to the prompt communication device 100B.

[0101] Next, an example of image generation in the prompt communication system of this embodiment will be described with reference to Fig. 13. Fig. 13(A) shows an example of a prompt for image generation acquired by the prompt acquisition unit 1101A. When the prompt / image conversion unit 1104A of the prompt communication device 100A converts this prompt for image generation A into image data, the converted image data will look, for example, as shown in Fig. 13(B). At this time, it is assumed that the prompt interpretation degree for "Akita dog" is 0% (or a value close to 0%). On the other hand, when the prompt / image conversion unit 1103B of the prompt communication device 100B converts this into image data using the same prompt for image generation A, the image data will look like Fig. 13(C). In other words, because "Akita dog" cannot be interpreted, image data may be generated in which "Akita dog" is replaced with another dog breed, "Siberian Husky."

[0102] Furthermore, suppose the subject of image generation prompt A is "Akita dog," and the prompt interpretation degree judgment criterion for "Akita dog" is set to 100% (or a value close to 100%). Using this image generation prompt, the prompt interpretation degree judgment unit 1101B judges the prompt interpretation degree and finds that "Akita dog" does not satisfy the criterion. In this case, the prompt communication device 100B transmits the prompt interpretation degree judgment result to the prompt communication device 100A. The prompt interpretation degree judgment result may include information indicating whether the criterion is satisfied. Alternatively, the prompt interpretation degree judgment result may include information indicating a portion of the prompt (e.g., "Akita dog") whose interpretation degree is lower than a predetermined criterion. The prompt interpretation degree judgment result may include, for example, information indicating a portion of the prompt (e.g., "Akita dog") that does not satisfy the criterion set by the prompt communication device 100A. The prompt interpretation degree judgment result may also include "Akita dog," which did not satisfy the criterion, and information indicating its interpretation degree (e.g., 0%). Alternatively, the prompt interpretation degree determination result may include "Akita dog," which did not satisfy the determination criterion, and the ratio of the interpretation degree to the determination criterion (e.g., 0 / 100). In this way, based on the prompt interpretation degree determination result indicating the prompt portion "Akita dog" with a low interpretation degree, the prompt communication device 100A can understand from the determination result that the prompt communication device 100B was unable to interpret "Akita dog." Furthermore, if the prompt interpretation degree determination result includes the interpretation degree (or the ratio of the interpretation degree to the determination criterion) in prompt communication device B, the explanation added to the prompt may be adjusted in accordance with the interpretation degree (or the ratio) when modifying the prompt.

[0103] The prompt communication device 100A modifies the image generation prompt A based on the prompt interpretation degree determination result. For example, as shown in Figure 13(D), the image generation prompt is modified to further describe the characteristics of the "Akita dog." The modified image generation prompt D is then transmitted again to the prompt communication device 100B.

[0104] The prompt / image converter 1103B of the prompt communication device 100B converts the received image generation prompt D into image data. The converted image data may look like that shown in Figure 13(E), for example. That is, image data that is closer to reproducing the image data in Figure 13(B) than the image data in Figure 13(C) is generated.

[0105] In this embodiment, prompt communication is repeated between prompt communication devices A and B while modifying the image generation prompt to improve the degree of interpretation. In this way, the image intended by prompt communication device 100A can be generated by prompt communication device 100B.

[0106] The prompt communication device 100B may update the prompt / image conversion model using the image generation prompt modified by the prompt communication device 100A or the image data transmitted by the prompt communication device 100A. As a result, the prompt communication device 100B becomes able to interpret "Akita dog."

[0107] As described above, in this embodiment, image generation prompts are communicated between devices, and one device modifies the image generation prompt based on the other device's level of prompt interpretation for the prompt so as to improve the other device's level of interpretation. For example, the device modifying the prompt modifies the image generation prompt by adding a description to a portion of the explanation with a low level of interpretation. This allows the receiving device to generate an image that closely resembles the image intended by the prompt provider.

[0108] In the above embodiment, an example of a prompt communication system has been described in which prompts for image generation are transmitted and received between the prompt communication device 100A and the prompt communication device 100B. However, prompt communication is not limited to the above example. That is, the other party of prompt communication with the prompt communication device 100A may be a prompt communication device other than the prompt communication device 100B. Conversely, the other party of prompt communication with the prompt communication device 100B may be a prompt communication device other than the prompt communication device 100A. For example, an external server may be provided that provides (shares) prompts for image generation to multiple prompt communication devices. In this case, the prompt communication device 100A and the prompt communication device 100B may obtain prompts for image generation from the external server. Then, the prompt communication device 100A may modify the prompts for image generation so that the degree of interpretation of the prompt communication device 100B is improved based on information about the degree of interpretation of the prompt by the prompt communication device 100B. This also makes it possible to generate an image that is closer to the image intended by the prompt provider.

[0109] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0110] (Disclosure of the present specification) The disclosure of this specification includes the following image processing device, image processing method, system, and program. (Item 1) an acquisition means for acquiring, from a predetermined external device, information on the degree of interpretation by the predetermined external device of a prompt including an explanation of an image to be generated by the generative model; a modifying means for modifying the prompt based on the information about the degree of interpretation; a transmitting means for transmitting the modified prompt to the predetermined external device via a communication means; The image processing device is characterized in that the correction means corrects the explanation of the prompt so that the explanation reflected in the image generated by the generative model increases. (Item 2) 2. The image processing device according to item 1, wherein the information about the degree of interpretation includes information indicating a part of the description where the degree to which the description is reflected in the image does not meet a predetermined standard. (Item 3) 3. The image processing device according to item 2, wherein the correction means corrects the prompt by adding a description for a part of the description that does not satisfy the predetermined criteria. (Item 4) 3. The image processing device according to item 2, wherein the correction means corrects the prompt by replacing a part of the explanation that does not satisfy the predetermined criteria with another explanation. (Item 5) a generating means for inputting the prompt into a generative model to generate an image; 3. The image processing device according to item 2, further comprising: a control means for controlling the generation means to generate an image from the prompt in response to a specific condition being satisfied, and for controlling the generated image to be transmitted to the predetermined external device. (Item 6) Item 6. The image processing device according to item 5, characterized in that the specific condition is satisfied when at least one of the following occurs: a request is received from the specified external device to send an image corresponding to the prompt; and the number of parts of the description whose degree of reflection in the image does not satisfy the specified criterion exceeds a specified number. (Item 7) the image processing device repeatedly performs operations including obtaining information about the interpretation level, modifying the prompt, and transmitting the modified prompt; 6. The image processing device according to item 5, wherein the specific condition is met when, when the operation is repeated, a part of the description that does not satisfy the predetermined criterion exists more than a predetermined number of times. (Item 8) 8. The image processing device according to any one of items 1 to 7, wherein the degree of interpretation of the prompt is determined in units of at least one of words, phrases, clauses, and sentences that make up the explanation. (Item 9) the prompt further includes criteria for determining an interpretability of the prompt; The image processing device includes: 9. The image processing device according to any one of items 1 to 8, further comprising a setting means for setting the criteria for at least one of a word, a phrase, a clause, and a sentence that constitute the description. (Item 10) 10. The image processing device according to item 9, wherein the setting means changes the determination criteria according to a history of communication of the prompt with the predetermined external device. (Item 11) A determination means for determining a degree of interpretation of a prompt including an explanation about an image to be generated by the generative model; and control means for controlling the provision of information about the interpretation level to a predetermined external device via communication means; The image processing device is characterized in that, after providing information about the degree of interpretation, the control means obtains from the specified external device a prompt in which the explanation is modified so that the explanation reflected in the image generated by the generative model increases. (Item 12) 12. The image processing device according to item 11, further comprising a generating means for inputting the prompt with the modified explanation into a generative model to generate an image. (Item 13) Item 13. The image processing device according to item 11 or 12, characterized in that the information about the degree of interpretation includes information indicating a part of the description where the degree to which the description is reflected in the image does not meet a predetermined standard. (Item 14) Item 14. The image processing device of item 13, wherein the prompt with the modified explanation is modified by adding a description of a part of the explanation that does not meet the predetermined criteria. (Item 15) Item 14. The image processing device of item 13, wherein the prompt with the modified explanation is modified by replacing a part of the explanation that does not meet the predetermined criteria with another explanation. (Item 16) Item 14. The image processing device according to item 13, wherein the control means controls the image generated from the prompt to be acquired from the predetermined external device in response to a specific condition being met. (Item 17) Item 17. The image processing device according to item 16, characterized in that the specific condition is satisfied when at least one of the following occurs: an instruction is received from the specified external device requesting an image corresponding to the prompt; or the number of parts of the description whose degree of reflection in the image does not satisfy the specified criterion exceeds a specified number. (Item 18) the image processor repeatedly performs operations including providing information about the interpretation level and obtaining a modified version of the prompt; Item 17. The image processing device according to item 16, wherein the specific condition is met if, when the operation is repeated, a part of the description that does not satisfy the predetermined criterion exists more than a predetermined number of times. (Item 19) 19. The image processing device according to any one of items 11 to 18, wherein the degree of interpretation of the prompt is determined in units of at least one of words, phrases, clauses, and sentences that make up the explanation. (Item 20) the prompt further includes criteria for determining an interpretability of the prompt; 20. The image processing device according to any one of items 11 to 19, wherein the criteria are set by the predetermined external device in units of at least one of words, phrases, clauses, and sentences that constitute the description. (Item 21) 21. The image processing device according to item 20, wherein the determination criteria are changed according to a history of communication between the prompt and the predetermined external device. (Item 22) A system including a first image processing device and a second image processing device, The first image processing device, A determination means for determining a degree of interpretation of a prompt including an explanation about an image to be generated by the generative model; a control means for controlling the second image processing device to provide information about the interpretation degree to the second image processing device via a communication means; The second image processing device, an acquisition means for acquiring information on the interpretation degree from the first image processing device; a modifying means for modifying the prompt based on the information about the degree of interpretation; a transmitting means for transmitting the modified prompt to the first image processing device via a communication means; The system is characterized in that the correction means corrects the explanation of the prompt so that the explanation reflected in the image generated by the generative model increases. (Item 23) An image processing method executed in an image processing device, obtaining, from a predetermined external device, information about the degree to which the predetermined external device interprets a prompt including a description of an image to be generated by the generative model; modifying the prompt based on the information about the degree of interpretation; transmitting the modified prompt to the predetermined external device via a communication means; An image processing method, characterized in that the modifying step modifies the explanation of the prompt so that the explanation reflected in the image generated by the generative model increases. (Item 24) An image processing method executed in an image processing device, determining a degree of interpretation of a prompt including a description of the image to be generated by the generative model; and a step of controlling the information about the interpretation level to be provided to a predetermined external device via a communication means, The image processing method is characterized in that the controlling step includes, after providing information about the degree of interpretation, obtaining a prompt from the specified external device in which the explanation is modified so that the explanation reflected in the image generated by the generative model increases. (Item 25) A program for causing a computer to function as each of the means of the image processing device described in any one of items 1 to 21.

[0111] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0112] 100A...prompt communication device, 100B...prompt communication device, 1101A...prompt acquisition unit, 1102A...prompt correction unit, 1103A...communication unit, 1104A...prompt / image conversion unit, 1101B...communication unit, 1102B...prompt interpretation degree determination unit, 1103B...prompt / image conversion unit, 1104B...image data display unit

Claims

1. an acquisition means for acquiring, from a predetermined external device, information on the degree of interpretation by the predetermined external device of a prompt including an explanation of an image to be generated by the generative model; a modifying means for modifying the prompt based on the information about the degree of interpretation; a transmitting means for transmitting the modified prompt to the predetermined external device via a communication means; The image processing device is characterized in that the correction means corrects the explanation of the prompt so that the explanation reflected in the image generated by the generative model increases.

2. The image processing device according to claim 1 , wherein the information about the degree of interpretation includes information indicating a part of the description where the degree to which the description is reflected in the image does not satisfy a predetermined standard.

3. 3. The image processing apparatus according to claim 2, wherein the correcting means corrects the prompt by adding a description for a part of the explanation that does not satisfy the predetermined criteria.

4. 3. The image processing apparatus according to claim 2, wherein the modifying means modifies the prompt by replacing a part of the explanation that does not satisfy the predetermined criteria with another explanation.

5. a generating means for inputting the prompt into a generative model to generate an image; 3. The image processing device according to claim 2, further comprising: a control unit that controls the generation unit to generate an image from the prompt in response to a specific condition being satisfied, and to transmit the generated image to the predetermined external device.

6. The image processing device described in claim 5, characterized in that the specific condition is satisfied at least when a request to send an image corresponding to the prompt is received from the specified external device, or when the number of parts of the description whose degree of reflection in the image does not meet the specified standard exceeds a specified number.

7. the image processing device repeatedly performs operations including obtaining information about the interpretation level, modifying the prompt, and transmitting the modified prompt; The image processing device according to claim 5 , wherein the specific condition is satisfied when, when the operation is repeated, a part of the description that does not satisfy the predetermined criterion exists more than a predetermined number of times.

8. The image processing device according to claim 1 , wherein the degree of interpretation of the prompt is determined in units of at least one of a word, a phrase, a clause, and a sentence that constitute the explanation.

9. the prompt further includes criteria for determining an interpretability of the prompt; The image processing device includes:

2. The image processing apparatus according to claim 1, further comprising setting means for setting the criteria for judgment in units of at least one of a word, a phrase, a clause, and a sentence that constitute the description.

10. 10. The image processing apparatus according to claim 9, wherein the setting means changes the determination criteria in accordance with a history of communication of the prompt with the predetermined external device.

11. A determination means for determining a degree of interpretation of a prompt including an explanation about an image to be generated by the generative model; and control means for controlling the provision of information about the interpretation level to a predetermined external device via communication means; The image processing device is characterized in that, after providing information about the degree of interpretation, the control means obtains from the specified external device a prompt in which the explanation is modified so that the explanation reflected in the image generated by the generative model increases.

12. 12. The image processing apparatus according to claim 11, further comprising a generating means for inputting the prompt with the modified explanation into a generative model to generate an image.

13. 12. The image processing device according to claim 11, wherein the information about the degree of interpretation includes information indicating a part of the description where the degree to which the description is reflected in the image does not satisfy a predetermined standard.

14. The image processing device of claim 13 , wherein the prompt with the modified explanation is modified by adding a description of a portion of the explanation that does not meet the predetermined criteria.

15. 14. The image processing device of claim 13, wherein the prompt with the modified explanation is modified by replacing a portion of the explanation that does not meet the predetermined criteria with another explanation.

16. 14. The image processing device according to claim 13, wherein the control means controls the image generated from the prompt to be acquired from the predetermined external device in response to a specific condition being satisfied.

17. The image processing device described in claim 16, characterized in that the specific condition is satisfied when at least one of the following occurs: an instruction is received from the specified external device requesting an image corresponding to the prompt; and the number of parts of the description whose degree of reflection in the image does not meet the specified standard exceeds a specified number.

18. the image processor repeatedly performs operations including providing information about the interpretation level and obtaining a modified version of the prompt; 17. The image processing device according to claim 16, wherein the particular condition is met if, when the action is repeated, a part of the description that does not satisfy the predetermined criterion exists more than a predetermined number of times.

19. The image processing device according to claim 11, wherein the degree of interpretation of the prompt is determined in units of at least one of a word, a phrase, a clause, and a sentence that constitute the explanation.

20. the prompt further includes criteria for determining an interpretability of the prompt; 12. The image processing apparatus according to claim 11, wherein the criteria are set by the predetermined external device in units of at least one of words, phrases, clauses, and sentences that make up the description.

21. 21. The image processing apparatus according to claim 20, wherein the determination criteria are changed according to a history of communication between the prompt and the predetermined external device.

22. A system including a first image processing device and a second image processing device, The first image processing device, A determination means for determining a degree of interpretation of a prompt including an explanation about an image to be generated by the generative model; a control means for controlling the second image processing device to provide information about the interpretation degree to the second image processing device via a communication means; The second image processing device, an acquisition means for acquiring information on the interpretation degree from the first image processing device; a modifying means for modifying the prompt based on the information about the degree of interpretation; a transmitting means for transmitting the modified prompt to the first image processing device via a communication means; The system is characterized in that the correction means corrects the explanation of the prompt so that the explanation reflected in the image generated by the generative model increases.

23. An image processing method executed in an image processing device, obtaining, from a predetermined external device, information about the degree to which the predetermined external device interprets a prompt including a description of an image to be generated by the generative model; modifying the prompt based on the information about the degree of interpretation; transmitting the modified prompt to the predetermined external device via a communication means; An image processing method, characterized in that the modifying step modifies the explanation of the prompt so that the explanation reflected in the image generated by the generative model increases.

24. An image processing method executed in an image processing device, determining a degree of interpretation of a prompt including a description of the image to be generated by the generative model; and a step of controlling the information about the interpretation level to be provided to a predetermined external device via a communication means, The image processing method is characterized in that the controlling step includes, after providing information about the degree of interpretation, obtaining a prompt from the specified external device in which the explanation is modified so that the explanation reflected in the image generated by the generative model increases.

25. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 21.

Images