Image processing device, image processing method, program, and system

Abstract

An image processing device according to the present disclosure comprises: an acquisition means for acquiring, from a predetermined external device, information regarding a degree of interpretation by the predetermined external device with respect to a prompt including a description regarding an image to be generated by a generation model; a correction means for correcting the prompt on the basis of the information regarding the degree of interpretation; and a transmission means for transmitting the corrected prompt to the predetermined external device via a communication means. Here, the correction means corrects the description of the prompt such that the description reflected in the image generated by the generation model develops.

Description

Image processing device, image processing method, program, and system

[0001] The present invention relates to an image processing device, an image processing method, a program, and a system.

[0002] In recent years, there has been remarkable progress in technology for generating images using models trained by machine learning. Technology for editing or generating images by inputting textual instructions for image generation, called "prompts," into a machine learning model for image generation (also simply referred to as an image generation model) is becoming widespread. Patent Document 1 discloses a technology for changing the facial expression of a subject as desired by inputting an image and text related to emotions.

[0003] Japanese Patent Application Laid-Open No. 2022-175923

[0004] However, if the data used for learning or the degree of learning differs between image generation models, differences in the interpretation of the prompt may occur even when the same prompt is input to a model with the same structure. Therefore, when the same prompt is used to generate images using the image generation models of multiple devices, the prompt may be interpreted differently on each device, and some devices may generate images that differ from the intention of the prompt provider.

[0005] The present invention has been made in view of the above-mentioned problems, and realizes a technology that, when generating an image from a prompt, can generate an image that is close to the image intended by the prompt provider.

[0006] In order to solve this problem, for example, an image processing device of the present invention has the following configuration: namely, it has an acquisition means that acquires, from a predetermined external device, information about the degree of interpretation by the predetermined external device of a prompt that includes an explanation about an image to be generated by the generative model, a correction means that corrects the prompt based on the information about the degree of interpretation, and a transmission means that transmits the corrected prompt to the predetermined external device via a communication means, wherein the correction means corrects the explanation of the prompt so as to increase the amount of the explanation reflected in the image generated by the generative model.

[0007] According to the present invention, when an image is generated from a prompt, it is possible to generate an image that is close to the image intended by the person providing the prompt. Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar components are designated by the same reference numerals.

[0008] The accompanying drawings are included in the specification, constitute a part thereof, illustrate embodiments of the present invention, and together with the description thereof, are used to explain the principles of the present invention.

[0023] Figures showing examples of hardware configurations of each embodiment: Block diagram showing an example of the functional configuration of a prompt communication device of embodiment 1; Flowchart showing the operation of image generation processing in the prompt communication device of embodiment 1; Flowchart showing the operation of prompt interpretation degree calculation processing in embodiment 1; Figure explaining image generation by the prompt communication device of embodiment 1; Figure explaining image generation by the prompt communication device of embodiment 1; Figure explaining image generation by the prompt communication device of embodiment 1; Figure explaining image generation by the prompt communication device of embodiment 1; Figure showing an example of image display by the prompt communication device of embodiment 1; Flowchart showing the operation when a weight value is used in the prompt interpretation degree calculation processing in embodiment 1; Block diagram showing an example of the functional configuration of a prompt communication device of embodiment 2; Flowchart showing the operation of image generation processing in the prompt communication device of embodiment 2; Prompt communication in embodiment 2 FIG. 1 is a diagram illustrating image generation by a prompt communication device according to a second embodiment. FIG. 2 is a diagram illustrating image generation by a prompt communication device according to a second embodiment. FIG. 3 is a diagram illustrating an example of the functional configuration of a prompt communication system according to a third embodiment. FIG. 4 is a flowchart illustrating the operation of initial processing in a prompt communication device 100A according to a third embodiment. FIG. 5 is a flowchart illustrating the operation of steady-state processing in a prompt communication device 100B according to a third embodiment. FIG. 6 is a flowchart illustrating the operation of steady-state processing in a prompt communication device 100A according to a third embodiment. FIG. 7 is a diagram illustrating image generation by a prompt communication system according to a third embodiment.

[0009] (Embodiment 1) Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings. Note that the following embodiment does not limit the invention according to the claims. Although multiple features are described in the embodiment, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the following, an example of an image processing device in each embodiment will be described, in which a communication device (also referred to as a prompt communication device) capable of handling prompts for generating an image (also referred to simply as image generation prompts; details will be described later) is used. This embodiment can be applied to other devices capable of handling image generation prompts. These devices may include, for example, digital cameras, mobile phones including smartphones, game consoles, personal computers, tablet terminals, and other wearable information terminals.

[0011] <Overview of the Use of Image Generation Prompts> Many images are captured with devices such as digital cameras and smartphones. However, capturing images often requires advanced photography skills to capture the photographer's experiences and emotions, and it is impossible to capture events that did not exist at the time of capture. As a result, captured images alone may not adequately convey the photographer's intended message. Instead of capturing images, technologies are now available that allow users to input image generation prompts into an image generation model to generate desired images. In particular, advances have been made in multimodal AI technology, which can simultaneously process multiple different types of information, such as text, images, audio, and video. Multimodal AI technology now enables the generation of images that reproduce the content of text and the generation of text that describes the scene content of images with high accuracy using a single generation model, enabling bidirectional conversion between images and text.

[0012] The image generation prompt is data that can be input to the image generation model and includes at least text information composed of a sentence or a list of words. An example of the image generation prompt, described below, can be written as follows: "The image shows a dog running on a lawn. The dog is an Akita with brown fur." ​​In other words, the image generation prompt includes a description of the image to be generated by the image generation model, written in natural language. The description of the image in the image generation prompt may include, for example, a word that represents the object (subject) included in the image to be generated, or a description of the object. The image generation prompt can be used as an instruction for generating an image to the image generation model.

[0013] Because prompts are written in natural language, they can easily describe emotions and atmospheres that are difficult to express in images obtained by photography. Furthermore, by using a generative model, high-resolution images can be generated that are faithful to the information contained in the prompt. In other words, by sharing prompts with others and using images generated from the prompts, it becomes easier to convey experiences and emotions to others. On the other hand, as described above, when the same prompt is used to generate images using the image generation models of multiple devices, the prompts may be interpreted differently on each device. Furthermore, some devices may generate images that differ from the intent of the prompt provider. Therefore, the following embodiments specifically describe a technique that allows a user to easily grasp the degree to which a device has interpreted a prompt, and a technique that, when generating an image from a prompt, generates an image that closely resembles the intent of the prompt provider.

[0014] 1 shows an example of the hardware configuration of a prompt communication device according to this embodiment. The prompt communication device 100 includes, for example, a system bus 101, a CPU 102, a ROM 103, a RAM 104, a HDD 105, a GPU 106, an input device 107, a communication device 108, and a display device 109. Each component of the prompt communication device 100 is connected to the system bus 101.

[0015] The CPU 102 includes one or more processors, such as a CPU (Central Processing Unit), and controls each function of the prompt communication device 100 by loading computer programs stored in the ROM 103 or the HDD 105 into the RAM 104 and executing them. The ROM 103 includes a non-volatile storage medium, such as a semiconductor memory, and stores, for example, programs executed by the CPU 102 and necessary data. The RAM 104 includes a volatile storage medium, such as a semiconductor memory, and temporarily stores, for example, calculation results of the CPU 102. The HDD 105 includes a hard disk drive and stores, for example, computer programs executed by the CPU 102 and the processing results thereof. In this example, the prompt communication device 100 is described as having a hard disk, but the prompt communication device 100 may also include a storage medium such as an SSD instead of a hard disk.

[0016] The GPU (Graphics Processing Unit) 106 includes one or more processors and can execute, for example, part or all of the processing in the learning stage and the inference stage of the image generation model. Compared to a CPU, a GPU can process more data in parallel. The processing in the learning stage and the inference stage of the image generation model may involve repeated calculations using a large-scale neural network including a large number of neurons, so it is effective to perform the processing using a GPU.

[0017] The input device 107 includes operation members such as buttons and a touch panel that accept operation inputs to the prompt communication device 100. The communication device 108 communicates with external devices via a network such as the Internet. The display device 109 includes a display panel such as an OLED, and displays, for example, images and text information.

[0018] <Example of Functional Configuration of Prompt Communication Device> Figure 2 shows an example of the functional configuration of the prompt communication device 100 in this embodiment. Note that the example functional configuration shown in Figure 2 focuses on the configuration for generating an image using an image generation prompt and displaying the generated image. Therefore, the configuration of the prompt communication device 100 is not limited to the configuration shown in Figure 2. Some or all of the example functional configuration shown in Figure 2 may be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program.

[0019] The prompt communication device 100 includes, for example, a prompt acquisition unit 201 , a prompt interpretation degree calculation unit 202 , a prompt search unit 203 , a prompt database 204 , a prompt / image conversion unit 205 , and an image data display unit 206 .

[0020] The prompt acquisition unit 201 acquires an image generation prompt from an external device via the communication device 108 and outputs it to the RAM 104. Note that in this embodiment, an example will be described in which an image generation prompt generated by an external device is acquired from the external device via the communication device 108. However, the configuration of the prompt acquisition unit 201 is not limited to this, and for example, the prompt acquisition unit 201 may acquire an image generation prompt recorded on an external server or cloud via communication with an external network. Alternatively, the prompt acquisition unit 201 may access an external storage device such as a USB memory or an external HDD and acquire an image generation prompt recorded in the external storage device.

[0021] The prompt interpretation degree calculation unit 202 calculates the degree of interpretation of the input image generation prompt. The explanation and calculation method of the interpretation degree of the image generation prompt (also simply referred to as the prompt interpretation degree) will be described later. The prompt interpretation degree calculation unit 202 is composed of, for example, a prompt search unit 203 and a prompt database 204. The prompt search unit 203 searches the prompt database 204 for the content described in the input image generation prompt. The prompt database 204 is stored in the HDD 105 and stores expressions that can be reflected in images by a generation model. In other words, the prompt database 204 is a group of information on prompts that can be converted into image data by the prompt / image conversion unit 205, which will be described later. The prompt interpretation degree calculation unit 202 calculates and outputs the prompt interpretation degree based on the search results of the prompt database 204.

[0022] The prompt / image conversion unit 205 reads an image generation prompt from the RAM 104 and converts the image generation prompt into image data (i.e., generates image data corresponding to the image generation prompt). The prompt / image conversion unit 205 outputs the obtained image data to the RAM 104. The conversion into image data is realized, for example, by the GPU 106 reading a prompt / image conversion model from the ROM 103 and performing arithmetic processing. The prompt / image conversion model is, for example, a machine learning model that has learned the relationship between various image generation prompts and images (i.e., has learned to generate images corresponding to various image generation prompts from the various image generation prompts). Note that known techniques related to image generation models can be applied to the processing in the learning and inference stages of the prompt / image conversion model, and therefore a description thereof will be omitted in this embodiment. Furthermore, this embodiment will be described taking as an example a case where the processing of the prompt / image conversion unit 205 is performed in the prompt communication device 100. However, the function of the prompt / image conversion unit 205 may be provided externally, and the processing of the prompt / image conversion unit 205 may be realized, for example, on an external server or cloud.

[0023] The image data display unit 206 displays various image data and text information on the display device 109. In this embodiment, at least the image data converted by the prompt / image conversion unit 205 is displayed.

[0024] <Image Generation Processing Operation Using Image Generation Prompt> The image generation processing operation using the image generation prompt in the prompt communication device 100 of this embodiment will be described with reference to FIGS. 3 to 5D.

[0025] Fig. 3 shows the overall operation of the image generation process using the image generation prompt, and Fig. 4 shows the operation of the prompt interpretation degree calculation process of this embodiment. This process can be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program to cause the various units shown in Fig. 2 to function.

[0026] In S301, the prompt acquisition unit 201 acquires an image generation prompt from an external device via the communication device 108 and outputs it to the RAM 104. Fig. 5A shows an example of the acquired image generation prompt. The image generation prompt shown in Fig. 5A includes information such as "The image shows a dog running on a lawn. The dog is an Akita dog with brown fur," as an explanation of the image to be generated by the image generation model.

[0027] In S302, the prompt interpretation degree calculation unit 202 reads the image generation prompt acquired in S301 from the RAM 104, calculates a prompt interpretation degree, and outputs the degree to the RAM 104. Here, the operation of the prompt interpretation degree calculation process will be described with reference to FIG.

[0028] <Operation of Prompt Interpretation Degree Calculation Process> Similar to the image generation process, the prompt interpretation degree calculation process can be realized by, for example, the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program to cause the various units shown in FIG. 2 to function. The prompt interpretation degree calculation process is started when the above-described step S302 is executed. Note that the following description uses an example in which a prompt interpretation degree is calculated, but as long as the prompt interpretation degree can be determined, the value may be obtained using a table, not limited to when the prompt interpretation degree is calculated.

[0029] In S401, the prompt search unit 203 divides the input image generation prompt into units for calculating the degree of interpretation. The prompt interpretation degree may be calculated for each word, phrase, clause, or sentence that constitutes the image generation prompt (i.e., the description of the image). Figure 5B shows the divided image generation prompt and the modifying relationships between the words. For example, the prompt search unit 203 divides the image generation prompt shown in Figure 5A into units of words such as "grass," "run," "one," "dog," "brown," and "Akita dog" as shown in Figure 5B.

[0030] In S402, the prompt search unit 203 uses the prompt database 204 to search the input image generation prompt for the units divided in S401. The prompt database 204 stores a group of prompt (text) information that the prompt / image conversion unit 205 can convert into image data. The prompt / image conversion unit 205 cannot convert text information that does not exist in the prompt database 204 into image data. Therefore, text information that does not exist in the prompt database 204 is content that the prompt communication device 100 cannot interpret (cannot reflect in an image). Therefore, the prompt communication device 100 can use the prompt database 204 to search for units into which the image generation prompt has been divided (i.e., parts of the description of the image), thereby determining whether the units can be reflected in the image.

[0031] In this embodiment, we will explain an example in which, among the contents divided in S402, "Akita dog" does not exist in the prompt database 204, and therefore the target text cannot be found when the prompt search unit 203 searches for "Akita dog."

[0032] In S403, the prompt search unit 203 calculates the degree of interpretation of the image generation prompt. For example, the prompt search unit 203 calculates the degree of prompt interpretation for each prompt divided in S401 based on the prompt search results acquired in S402. The prompt search unit 203 outputs the calculated prompt interpretation degree to RAM 104. After outputting the prompt interpretation degree to RAM 104, the prompt search unit 203 ends this process and returns to the caller.

[0033] 5B shows the modifying relationships for the image generation prompts divided into words in S401. For example, information such as the breed of dog "Akita" and the coat color "brown" is qualitative information that indicates the appearance, attributes, and characteristics of the "dog." Information indicating movement and information about the number of "dogs," such as "one," is instantaneous information that describes the state of the "dog" over a specific period of time. All of this information is written to modify the "dog."

[0034] In this embodiment, the prompt interpretation degree represents, for example, the possibility of interpretation of a specific word or phrase, or the ratio of the possibility of interpretation of information modifying a specific word or phrase. For example, the prompt interpretation degree for "Akita dog" is 0% (or a value close to 0%). As shown in FIG. 5B , the image generation prompt in FIG. 5A describes four pieces of information modifying the word "dog": breed, coat color, movement, and number. If the breed "Akita dog" is determined to be ininterpretable, while the coat color "brown," the movement "running," and the number "one" are determined to be interpretable, the interpretation degree for the word "dog" is 3 / 4. Therefore, the prompt search unit 203 can calculate the interpretation degree for the word "dog" as 75%. In this way, the prompt interpretation degree represents, for example, the degree to which the explanation about the image is reflected in the image.

[0035] In this embodiment, an example of a method for calculating the prompt interpretation degree for a specific word or phrase has been described. For a clause or sentence, the interpretation degree may be defined as the ratio of the possible interpretations of the words or phrases that make up the clause or sentence. The above description also deals with an example in which all information is treated as a single unit. However, when calculating the prompt interpretation degree, weight values ​​set for each word, phrase, clause, or sentence in the image generation prompt may also be taken into account. Calculation of the prompt interpretation degree using weight values ​​will be described later.

[0036] 3 again, in S303, the prompt / image conversion unit 205 reads out the prompt for image generation acquired in S301 from the RAM 104, converts it into image data, and outputs it to the RAM 104.

[0037] If the image generation prompt in FIG. 5A is sufficiently interpreted (e.g., if the prompt interpretability is sufficiently high), the process in S303 is expected to generate image data such as that shown in FIG. 5C. If the prompt contains a word or phrase that cannot be interpreted, the image can be generated by, for example, ignoring the content or substituting an alternative word that fits the context. In this embodiment, for example, because "Akita dog" cannot be interpreted, the prompt / image conversion unit 205 generates image data by substituting "Siberian Husky," a different breed of dog from "Akita dog" (existing in the prompt database 204). For example, the prompt / image conversion unit 205 generates image data such as that shown in FIG. 5D. When substituting words, the prompt / image conversion unit 205 may acquire information on the type of modifier, such as breed, coat color, movement, or number, for each word. For example, the prompt / image conversion unit 205 may estimate the type of modifier of a word using a learning model. When substituting the uninterpretable word "Akita dog," the prompt / image conversion unit 205 may use the estimated type of modifier to search a database for words of the same type of modifier (e.g., dog breed) as the uninterpretable word, and substitute the translatable word "Siberian Husky."

[0038] In S304, the image data display unit 206 reads the image generation prompt, information about the prompt interpretation level, and image data obtained in S301 to S303 from the RAM 104 and displays them on the display device 109. The information about the interpretation level may be, for example, information indicating an unknown word in the prompt, such as "Akita dog." In other words, the information about the interpretation level may include, for example, a portion of the description of the image in the prompt (e.g., "Akita dog") that does not meet the criteria for its reflection in the image.

[0039] FIG. 6 shows an example of the image display displayed in S304. For example, the image data display unit 206 displays the image generation prompt in text format in area 601. The image data display unit 206 also displays information about the prompt interpretation level in area 602, separate from area 601. For example, if the prompt contains at least one of a word, phrase, clause, or sentence whose prompt interpretation level is below a predetermined threshold, the image data display unit 206 notifies the user by displaying the word, phrase, clause, or sentence in area 602. At this time, the image data display unit 206 can highlight the word, phrase, clause, or sentence displayed in area 602. The predetermined threshold is set to notify the user when the image data converted in S303 contains a subject with low reproducibility for the input image generation prompt. In this embodiment, as an example, the predetermined threshold is set so that the user is notified of content (e.g., a word) whose prompt interpretation level is 0%. As a result, the image data display unit 206 displays "Akita dog" in area 602. Furthermore, the image data display unit 206 may display the image data converted in S303 in an area 603. The image data displayed in the area 603 is image data converted using only the prompt explanation that the generative model can reflect in the image. When the image data display unit 206 displays the image data, etc. on the display device 109, it ends the series of operations related to the image generation process.

[0040] In this way, the image generation process according to this embodiment calculates the degree of interpretation of the acquired image generation prompt and makes it possible to notify the user according to the calculation result. Note that in this embodiment, an example has been described in which a notification according to the degree of interpretation of the image generation prompt is displayed on the display device 109. However, the notification according to the degree of interpretation is not limited to the above example, and for example, a notification may be sent to an external device via the communication device 108.

[0041] <Operation When Weight Values ​​Are Used in Prompt Interpretation Degree Calculation Processing> As described above, the prompt interpretation degree calculation unit 202 may take into account the weight value of the image generation prompt when calculating the prompt interpretation degree. In this case, the weight value may be set in units of words, phrases, clauses, or sentences (i.e., parts of the description of the image) included in the image generation prompt. The prompt interpretation degree calculation processing when using weight values ​​will be described with reference to FIG. 7. Like the prompt interpretation degree calculation processing shown in FIG. 4, the prompt interpretation degree calculation processing shown in FIG. 7 can be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program to cause the various units shown in FIG. 2 to function. Note that processing similar to that shown in FIG. 4 is designated by the same reference numerals, and description thereof will be omitted.

[0042] In S401, the prompt search unit 203 divides the input image generation prompt into units for calculating the degree of interpretation, as described above. In S701, the prompt search unit 203 determines whether a weight value has been set for each division unit of the image generation prompt (i.e., a portion of the image description). For example, if a weight value is associated with each division unit of the image generation prompt, the prompt search unit 203 can determine that a weight value has been set. If the prompt search unit 203 determines that a weight value has been set, the process proceeds to S703; otherwise, the process proceeds to S702.

[0043] In S702, the prompt search unit 203 sets weight values ​​for each division unit of the image generation prompt. The weight value of the prompt may be determined, for example, based on the importance of the target word, clause, phrase, or sentence. For example, the prompt search unit 203 sets a high weight value for a topic word in the prompt, a word that may be the main subject of the image data generated by the prompt / image conversion unit 205, or a clause or sentence containing such a word. For example, the prompt search unit 203 sets a higher weight value for the topic words "lawn" and "dog" in the image generation prompt of FIG. 5A than for other information. Alternatively, the prompt search unit 203 may set a higher weight value for clauses or sentences containing "lawn" or "dog." Alternatively, the prompt search unit 203 may set a higher weight value for a unit (e.g., a word) that contains a lot of information for modifying a word representing a noun in the image generation prompt. For example, in the image generation prompt of FIG. 5A, there is no information modifying "lawn," but information such as the breed, coat color, and behavior of "dog" is described. In this case, the prompt search unit 203 assigns a higher weight value to "dog" than to "grass." That is, when weighting the image generation prompt shown in FIG. 5A , the prompt search unit 203 assigns the highest weight value to "dog," followed by "grass." The prompt search unit 203 then assigns the lowest weight value to information that modifies "dog," such as "one," "brown," and "Akita dog." It should be noted that known natural language analysis techniques and trained classification models can be used to estimate which of the division units (e.g., words) of the image generation prompt are the subject, which may be the main subject, and which are modifiers.

[0044] In the present embodiment, an example has been described in which some of the image generation prompts are reflected in the generated image (hereinafter referred to as positive prompts). However, there are also cases in which some of the image generation prompts are not reflected in the generated image (hereinafter referred to as negative prompts). In this case, the prompt search unit 203 may set a weight value for words, phrases, clauses, or sentences belonging to negative prompts higher than the weight value set for positive prompts, so as to prioritize the calculation of the degree of interpretation of negative prompts.

[0045] Furthermore, the prompt search unit 203 may assign different weight values ​​to the division units of the image generation prompt depending on specific syntax or emphasis expressions. For example, the prompt search unit 203 may assign a higher weight value to an earlier description order in the image generation prompt, or may assign a higher weight value according to the order of word appearance frequency. The prompt search unit 203 may assign a relatively higher weight value to expressions using exclamation marks or expressions that are emphasized by rhetoric.

[0046] In S703, the prompt search unit 203 searches the database 204 for division units of the image generation prompt in descending order of weight value. The search of the database 204 can be performed in the same manner as the process described above in S402. Note that the prompt search unit 203 may limit the prompts to be searched to those with a weight value equal to or greater than a predetermined threshold value. This can limit the number of searches performed and speed up the process.

[0047] In S704, the prompt search unit 203 calculates the degree of interpretation for each divided image generation prompt based on the prompt search results acquired in S703. In S704, the prompt search unit 203 calculates the degree of interpretation by assigning a weight to the ratio of the likelihood of interpretation to information modifying a specific word or phrase. For example, in the example shown in FIG. 5B , the image generation prompt of FIG. 5A describes four pieces of information modifying the word "dog." Furthermore, when the image generation prompt of FIG. 5A was acquired from an external device, the external device had already set a weight value for information about the dog breed at twice the weight of the other information (brown, running, one). If the dog breed "Akita" cannot be interpreted, the degree of interpretation for the word "dog" is 3 / 5. Therefore, the prompt search unit 203 calculates the degree of interpretation for the word "dog" to be 60%.

[0048] In this way, in the prompt interpretation degree calculation process, by using a weight value to calculate the interpretation degree of the acquired image generation prompt, it is possible to prioritize the interpretation degree for important information in the image generation prompt.

[0049] As described above, in this embodiment, the prompt communication device 100 determines the degree of interpretation of the acquired image generation prompt (i.e., the description of the image). Based on the degree of interpretation of the prompt, the prompt communication device 100 then displays on the display device 109 information about the degree of interpretation of the prompt (e.g., "Akita dog") whose reflection in the image does not satisfy a predetermined standard. This allows the user to understand the degree of interpretation of the prompt by the prompt device 100.

[0050] Second Embodiment Next, a second embodiment of the present invention will be described. This embodiment corresponds to a case in which the prompt communication device 100 described in the first embodiment does not have a prompt database 204. An example of a case in which the prompt database 204 is not included is when the internal processing of the image / prompt conversion model used in the prompt / image conversion process is kept confidential, making it impossible to grasp the internal processing. Alternatively, this also applies to a case in which the prompt communication device 100 does not have a prompt / image conversion unit 205 and the prompt / image conversion process is performed on an external server or cloud. Note that in this embodiment, the configuration of the prompt interpretation degree calculation unit and the processing performed by this unit differ from those in the first embodiment, but the other configurations are similar to those in the first embodiment. Therefore, components that are the same or substantially the same as those in the first embodiment are designated by the same reference numerals, and their description will be omitted.

[0051] 8 shows an example of the functional configuration of the prompt communication device 100 that calculates the degree of prompt interpretation in this embodiment. Furthermore, some or all of the functional configuration example shown in FIG. 8 may be realized by, for example, the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program.

[0052] The prompt communication device 100 includes, for example, a prompt acquisition unit 201 , a prompt interpretation degree calculation unit 801 , and an image data display unit 206 .

[0053] The prompt interpretation degree calculation unit 801 calculates the degree of interpretation of the input image generation prompt. In this embodiment, the prompt interpretation degree calculation unit 801 includes the prompt / image conversion unit 205, an image / prompt conversion unit 802, and a prompt comparison unit 803. The prompt / image conversion unit 205 is realized with the same configuration as in the first embodiment.

[0054] The image / prompt conversion unit 802 reads the image / prompt conversion model from the RAM 104 and generates a prompt for generating an image from the image data. The calculation processing of the image / prompt conversion unit 802 is performed, for example, by the GPU 106. The image / prompt conversion model is, for example, a machine learning model that has learned the relationship between various images and prompts for generating images (i.e., has learned to generate prompts corresponding to various images from the images). The conversion model used by the prompt / image conversion unit 205 and the image / prompt conversion unit 802 is a single generation model that enables bidirectional conversion between image data and prompts.

[0055] The prompt comparison unit 803 receives at least two image generation prompts and compares the contents of the prompts. Through the comparison, the prompt comparison unit 803 can identify words and the like that do not exist in one of the image generation prompts.

[0056] <Operation of Image Generation Processing Using Image Generation Prompt> Next, the image generation processing of this embodiment will be described with reference to Fig. 9. The image generation processing of this embodiment can be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100 executing a computer program to cause each unit of the prompt communication device 100 to function.

[0057] In S301, the prompt acquisition unit 201 acquires a prompt for image generation and outputs it to the RAM 104, as in the first embodiment. At this time, it is assumed that the image generation prompt A shown in FIG. 10A is acquired. In S303, the prompt / image conversion unit 205 reads the image generation prompt A from the RAM 104, converts it into image data, and outputs it to the RAM 104, as in the first embodiment. In this embodiment, the prompt / image conversion unit 205 converts the image generation prompt A into image data without being able to interpret the dog breed "Akita dog." As a result, it is assumed that image data such as that shown in FIG. 10B is generated. For example, image data is generated in which "Akita dog" is replaced with another dog breed, "Siberian Husky."

[0058] In S901, the image / prompt conversion unit 802 reads the image data converted in S303 from the RAM 104, converts it into image generation prompt B, and outputs it to the RAM 104. For example, the image / prompt conversion unit 802 converts image data ( FIG. 10B ) that was converted without being able to interpret the dog breed "Akita dog" into an image generation prompt. As a result, the image generation prompt B shown in FIG. 10C is generated. In other words, an image generation prompt that does not include information about the dog breed "Akita dog" is generated.

[0059] In S902, the prompt comparison unit 803 reads the image generation prompt A acquired in S301 and the image generation prompt B converted in S901 from the RAM 104. The prompt comparison unit 803 then compares the two read image generation prompts to calculate the degree of interpretation of the image generation prompt A.

[0060] When comparing image generation prompt A and image generation prompt B, the dog breed "Akita" is only included in image generation prompt A. Therefore, if image generation prompt B does not contain information identical to the words, phrases, clauses, or sentences that make up image generation prompt A, the prompt comparison unit 803 can determine that image generation prompt A is a prompt that includes information that cannot be interpreted.

[0061] In this way, by comparing the content of the input image generation prompt with the content converted into image data and then converted back into a prompt, it is possible to determine whether the image generation prompt can be interpreted. That is, the prompt comparison unit 803 can determine that the dog breed "Akita" is ininterpretable, but the coat color "brown," the movement "one," and the number "one" are interpretable. After identifying the ininterpretable content, the degree of interpretation can be calculated based on whether the content can be interpreted, as in the first embodiment.

[0062] In S304, the image data display unit 206 reads the image generation prompt A, the prompt interpretation degree, and the image data from the RAM 104, as in the first embodiment, and displays them on the display device 109 according to the calculation result of the prompt interpretation degree.

[0063] In this manner, the image generation process according to this embodiment converts the acquired prompt into an image (generates an image), converts the generated image into a prompt to reproduce the prompt, and then compares the acquired prompt with the reproduced prompt. Based on the comparison result of the prompts, the degree of interpretation of the acquired prompt is calculated. This also makes it possible to calculate the degree of interpretation of the prompt and notify the user according to the calculation result. In other words, the user can grasp the degree of interpretation of the prompt by the prompt device 100.

[0064] Third Embodiment Next, a third embodiment will be described. Fig. 11 shows an example of the functional configuration of a prompt communication device 100A and a prompt communication device 100B as an example of a prompt communication system for transmitting and receiving prompts between devices in this embodiment. The hardware configuration of the prompt communication device 100A and the prompt communication device 100B is the same as that of the prompt communication device 100 described above with reference to Fig. 1. Furthermore, for the convenience of distinguishing between the prompt communication devices, the prompt communication devices are given reference numerals with an A and a B added. Note that part or all of the example functional configuration shown in Fig. 11 may be realized, for example, by the CPU 102 or GPU 106 of the prompt communication device 100A or the prompt communication device 100B executing a computer program.

[0065] The prompt communication device 100A includes, for example, a prompt acquisition unit 1101A, a prompt correction unit 1102A, a communication unit 1103A, and a prompt / image conversion unit 1104A. The prompt communication device 100B includes, for example, a prompt interpretation degree determination unit 1101B, a communication unit 1102B, a prompt / image conversion unit 1103B, and an image data display unit 1104B.

[0066] The prompt acquisition unit 1101A acquires an image generation prompt from an external device via the communication device 108 and outputs it to the RAM 104 of its own device. In this embodiment, an example will be described in which the image generation prompt is acquired from the external device via the communication device 108. However, the configuration of the prompt acquisition unit 1101A is not limited to this. For example, the prompt acquisition unit 1101A may be provided with a communication unit for connecting to an external network and acquire an image generation prompt recorded on an external server or cloud. Alternatively, the prompt acquisition unit 1101A may have a connection unit for connecting to an external storage device and acquire an image generation prompt recorded in an external storage device such as a USB memory or an external HDD.

[0067] The prompt correction unit 1102A reads the image generation prompt and the prompt interpretation level for the image generation prompt from the RAM 104 of its own device, and performs a predetermined correction process on the image generation prompt based on the prompt interpretation level. The prompt correction unit 1102A outputs the image generation prompt that is the result of the correction process to the RAM 104 of its own device. Details of the correction process will be described later in the explanation of the prompt communication process.

[0068] The communication unit 1103A transmits and receives data to and from the prompt communication device 100B. For example, the communication unit 1103A reads data such as an image generation prompt from the RAM 104 of its own device and transmits it to the prompt communication device 100B, or receives data from the prompt communication device 100B and outputs it to the RAM 104 of its own device.

[0069] The communication unit 1102B transmits and receives data to and from the prompt communication device 100A. For example, the communication unit 1102B receives data from the prompt communication device 100A and outputs it to the RAM 104 of its own device, or reads data from the RAM 104 of its own device and transmits it to the prompt communication device 100A.

[0070] The prompt communication device 100A transmits image generation prompts output by the prompt acquisition unit 1101A and prompt correction unit 1102A to the prompt communication device 100B via the communication unit 1103A. The prompt communication device 100A may also transmit image data output by a prompt / image conversion unit 1104A (described below) to the prompt communication device 100B via the communication unit 1103A. The prompt communication device 100A also receives a prompt interpretation degree output by a prompt interpretation degree determination unit 1101B (described below) from the prompt communication device 100B via the communication unit 1103A.

[0071] Similarly, the prompt communication device 100B receives image generation prompts output by the prompt acquisition unit 1101A and the prompt correction unit 1102A from the prompt communication device 100A via the communication unit 1102B. The prompt communication device 100B may also receive image data output by the prompt / image conversion unit 1104A from the prompt communication device 100A. The prompt communication device 100B also transmits the prompt interpretation degree output by the prompt interpretation degree determination unit 1101B to the prompt communication device 100A via the communication unit 1102B.

[0072] Prompt / image conversion unit 1104A reads an image generation prompt from its own device's RAM 104, converts it to image data (i.e., generates image data according to the image generation prompt), and outputs it to its own device's RAM 104. Similarly, prompt / image conversion unit 1103B reads an image generation prompt from its own device's RAM 104, converts it to image data, and outputs it to its own device's RAM 104.

[0073] Conversion into image data can be performed in the same manner as in the first embodiment, and is realized, for example, by the GPU 106 reading out the prompt / image conversion model from the ROM 103 of its own device and performing arithmetic processing.

[0074] The prompt interpretation degree determination unit 1101B reads the image generation prompt from the RAM 104 of its own device and calculates the prompt interpretation degree. Additionally, the prompt interpretation degree determination unit 1101B determines the prompt interpretation degree using the prompt interpretation degree determination criteria read from the RAM 104 of its own device. The prompt interpretation degree determination unit 1101B then outputs the prompt interpretation degree determination result to the RAM 104. Note that the calculation of the prompt interpretation degree is assumed to be the same as the method described in either embodiment 1 or embodiment 2, and therefore a description of this method will be omitted in this embodiment. The determination of the prompt interpretation degree will be described in detail in the prompt communication processing described below.

[0075] The image data display unit 1104B displays various image data on the display device 109. In this embodiment, the image data display unit 1104B displays, for example, image data converted by the prompt / image conversion unit 1104A and the prompt / image conversion unit 1103B.

[0076] <Operation of Prompt Communication Processing> The prompt communication processing of the prompt communication device 100A and the prompt communication device 100B of this embodiment will be described with reference to FIGS. 12A to 12C.

[0077] Figures 12A to 12C illustrate the operation of the prompt communication process of this embodiment. The processes shown in Figures 12A and 12C can be realized by the respective units shown in Figure 11 when the CPU 102 or GPU 106 of the prompt communication device 100A executes a computer program. The process shown in Figure 12B can be realized by the respective units shown in Figure 11 when the CPU 102 or GPU 106 of the prompt communication device 100B executes a computer program. The processes shown in Figures 12A to 12C, which can be executed sequentially from the process shown in Figure 12A, can basically be executed when a communication instruction is received from the user, or at any time before or after that. However, even when a communication instruction is not received from the user, they can also be executed at regular intervals, for example, if the prompt communication devices 100A and 100B are always active and ready for communication.

[0078] <Initial Processing Operation of Prompt Communication Device 100A> Steps S1201 to S1203 in Fig. 12A show the initial processing operation of the prompt communication device 100A. Processing steps S1201 to S1203 are performed at the beginning of each prompt-related communication between the prompt communication device 100A and the prompt communication device 100B. The prompt communication device 100A then performs the routine processing operations of S1213 to S1218, which will be described later.

[0079] In S1201 , the prompt acquisition unit 1101A acquires an image generation prompt from an external device via the communication device 108 and outputs it to the RAM 104 .

[0080] In S1202, the prompt modification unit 1102A reads the image generation prompt acquired in S1201 from the RAM 104. Then, it sets a criterion for determining the degree of prompt interpretation for the image generation prompt. The prompt modification unit 1102A modifies the image generation prompt, for example, to set the criterion for determining the degree of prompt interpretation. The prompt modification unit 1102A outputs the modified image generation prompt to the RAM 104.

[0081] The prompt interpretation degree determination criterion is information accompanying the image generation prompt, and is a threshold value for the prompt interpretation degree calculated in step S1205, which will be described later. The prompt interpretation degree determination criterion can be set for each word, clause, phrase, or sentence that constitutes the image generation prompt.

[0082] The prompt interpretation level determination criteria can be determined or changed based on a communication history with the other device in the prompt communication. For example, if the prompt modification unit 1102A has a history of communication with the other device in the prompt communication, the prompt interpretation level for that word may satisfy the determination criteria. In this case, the prompt modification unit 1102A can either not set the prompt interpretation level determination criteria or set the determination criteria to a lower value (than when there is no communication history).

[0083] On the other hand, the prompt modification unit 1102A may set a higher prompt interpretation degree criterion for words representing the subject, words representing the main subject, or clauses or sentences containing these words among the image generation prompts than for other parts. Furthermore, the prompt interpretation degree criterion may be set higher for words representing nouns that contain a predetermined number or more of information that embodies the word, such as modifiers, or for clauses or sentences containing such words than for other parts. In this case, the other device may be requested to output a high interpretation degree for the word. In other words, the other device is required to have a generation model with high reproducibility for the word, or to generate an image that accurately reproduces the word by repeatedly receiving modified prompts.

[0084] In this embodiment, the prompt communication device 100A sets the prompt interpretation degree determination criteria. However, for example, the prompt interpretation degree determination criteria may be set in advance and stored in the RAM 104 of the prompt communication device 100A before the image generation prompt is acquired in S1201. Alternatively, the prompt interpretation degree determination criteria may be set in the prompt communication device 100B.

[0085] In S1203, the communication unit 1103A reads the image generation prompt corrected in S1202 from the RAM 104 and transmits it to the prompt communication device 100B. After transmitting the image generation prompt, the prompt communication device 100A ends the initial processing.

[0086] <Operation of Steady-State Processing of Prompt Communication Device 100B> Next, steady-state processing of the prompt communication device 100B will be described with reference to FIG. 12B.

[0087] In S1204, the communication unit 1102B receives the image generation prompt transmitted from the prompt communication device 100A in S1203 and S1216 (described later), and outputs it to its own device's RAM 104. In S1205, the prompt interpretation degree determination unit 1101B reads the image generation prompt received in S1204 from its own device's RAM 104, and calculates the prompt interpretation degree.

[0088] In S1206, the prompt interpretation degree determination unit 1101B determines the prompt interpretation degree. Specifically, the prompt interpretation degree determination unit 1101B reads the prompt interpretation degree determination criteria set in S1202 from the RAM 104 of its own device, and determines whether the prompt interpretation degree calculated in S1205 satisfies this determination criteria. The prompt interpretation degree determination unit 1101B then outputs the prompt interpretation degree determination result to the RAM 104 of its own device. If there is even one prompt that does not satisfy the determination criteria, the prompt interpretation degree determination unit 1101B determines that the prompt interpretation degree does not satisfy the determination criteria, and proceeds to S1207. On the other hand, if there is no prompt that does not satisfy the determination criteria, the prompt interpretation degree determination unit 1101B determines that the prompt interpretation degree satisfies the determination criteria, and proceeds to S1209.

[0089] In S1207, the prompt interpretation level determination unit 1101B determines whether image data will be received from the prompt communication device 100A (i.e., whether specific conditions for receiving image data are met). For example, the prompt interpretation level determination unit 1101B determines that image data will be received if the number of prompts that do not satisfy the determination criteria for the prompt interpretation level determination result determined in S1206 exceeds a predetermined number. Alternatively, the prompt interpretation level determination unit 1101B may determine that image data will be received if the number of times that at least one prompt does not satisfy the determination criteria exceeds a predetermined number when the operations of S1204 to S1209 are repeated. Alternatively, the prompt interpretation level determination unit 1101B may determine that image data will be received if it receives an instruction (e.g., an instruction to receive an image) from the prompt communication device 100A. Furthermore, the respective determination contents may be combined. If the prompt interpretation level determination unit 1101B determines that image data will be received, it proceeds to S1211; if not, it proceeds to S1208.

[0090] In S1208, the communication unit 1102B reads the determination result of the prompt interpretation level determined in S1206 from the RAM 104 of its own device, and transmits the determination result of the prompt interpretation level to the prompt communication device 100A.

[0091] Steps S1209 to S1210 are the process until the image data is displayed on the display device. In this embodiment, even if some prompts do not satisfy the criteria based on the prompt interpretation degree determination result determined in step S1206, the image generation prompt is displayed. However, the image generation prompt may be finally displayed when there are no more prompts that do not satisfy the criteria.

[0092] In S1209, the prompt / image conversion unit 1103B reads the image generation prompt received in S1204 from the RAM 104 of its own device, converts it into image data, and outputs it to the RAM 104 of its own device. In S1210, the image data display unit 1104B reads the image data converted in S1209 from the RAM 104 of its own device, and displays it on the display device 109. Once the prompt communication device 100B displays the image data, it ends one round of routine processing.

[0093] In S1211, the communication unit 1102B transmits a request for image data to the prompt communication device 100A. In S1212, the communication unit 1102B receives the image data transmitted from the prompt communication device 100A in S1218 (described later) and outputs the image data to the RAM 104 of its own device.

[0094] <Operation of Steady-State Processing of Prompt Communication Device 100A> Next, the operation of steady-state processing of the prompt communication device 100A will be described with reference to FIG. 12C.

[0095] In S1213, the prompt correction unit 1102A determines whether image data should be sent to the prompt communication device 100B (i.e., whether specific conditions for sending image data are met). For example, the prompt correction unit 1102A determines to send image data when it receives a request to send image data from the prompt communication device 100B. Alternatively, the prompt correction unit 1102A may determine to send image data when a predetermined number of prompts do not satisfy the determination criteria based on the prompt interpretation level determination results received in S1214 (described below). Alternatively, the prompt correction unit 1102A may determine to send image data when the number of times that at least one prompt does not satisfy the determination criteria exceeds a predetermined number when the operations of S1213 to S1216 are repeated. Furthermore, each determination may be combined. If the prompt correction unit 1102A determines to send image data, it proceeds to S1217. If it determines not to send image data, it proceeds to S1214.

[0096] In S1214, the communication unit 1103A receives the determination result of the prompt interpretation level sent from the prompt communication device 100B in S1208, and outputs it to the RAM 104 of its own device.

[0097] In S1215, the prompt correction unit 1102A reads the image generation prompt and the determination result of the prompt interpretation degree from the RAM 104 of its own device. Then, using the determination result of the prompt interpretation degree, the prompt correction unit 1102A corrects the image generation prompt so that the prompt interpretation degree is higher than the current prompt interpretation degree. In other words, the prompt correction unit 1102A corrects the image generation prompt so that the explanation about the image that is reflected in the generated image is increased. The prompt correction unit 1102A outputs the corrected image generation prompt to the RAM 104.

[0098] The image generation prompt can be modified, for example, by replacing the image generation prompt whose prompt interpretation level does not satisfy the determination criterion in S1206 with an alternative word. That is, the prompt modification unit 1102A can replace a portion of the image generation prompt (i.e., a portion of the description of the image) that does not satisfy the criterion with another description. Alternatively, the image generation prompt can be modified, for example, by adding a description to the image generation prompt whose prompt interpretation level does not satisfy the determination criterion in S1206 (i.e., adding a description of the image). For example, for the image generation prompt shown in FIG. 13A , a case will be described in which it is determined in S1206 that "Akita dog" does not satisfy the determination criterion. In this case, the prompt modification unit 1102A modifies the image generation prompt to add a further description about "Akita dog," as shown in FIG. 13D . The additional information can include, for example, "...it is a medium-sized dog," "its ears are triangular with drooping tips," and the like, and can also include a description that further describes the physical characteristics of the "Akita dog" that was determined not to satisfy the determination criterion.

[0099] In S1216, the communication unit 1103A reads the image generation prompt corrected in S1215 from RAM 104 and transmits it to the prompt communication device 100B. In this way, the prompt communication device 100B can generate an image from the corrected image generation prompt, and can generate an image with a higher degree of interpretation (i.e., an image that is closer to the image intended by the communication device 100A).

[0100] In S1217, (to send an image to the prompt communication device 100B) the prompt / image conversion unit 1104A reads the image generation prompt from its own RAM 104, converts it to image data, and outputs it to its own RAM 104. Note that the conversion from the image generation prompt to image data can be performed in the same manner as in embodiment 1. In S1218, the communication unit 1103A reads the image data converted in S1217 from its own RAM 104, and sends it to the prompt communication device 100B.

[0101] Next, an example of image generation in the prompt communication system of this embodiment will be described with reference to FIG. 13 . FIG. 13A shows an example of an image generation prompt acquired by the prompt acquisition unit 1101A. When the prompt / image conversion unit 1104A of the prompt communication device 100A converts the image data into image data using this image generation prompt A, the converted image data will look like, for example, FIG. B. In this case, assume that the prompt interpretation degree for "Akita dog" is 0% (or a value close to 0%). On the other hand, when the prompt / image conversion unit 1103B of the prompt communication device 100B converts the image data into image data using the same image generation prompt A, the image data will look like FIG. C. In other words, because "Akita dog" cannot be interpreted, image data may be generated in which "Akita dog" is replaced with another dog breed, "Siberian Husky."

[0102] Furthermore, suppose the subject of image generation prompt A is "Akita dog," and the prompt interpretation degree judgment standard for "Akita dog" is set to 100% (or a value close to 100%). Using this image generation prompt, the prompt interpretation degree judgment unit 1101B judges the prompt interpretation degree and finds that "Akita dog" does not satisfy the judgment standard. In this case, the prompt communication device 100B transmits the prompt interpretation degree judgment result to the prompt communication device 100A. The prompt interpretation degree judgment result may include information indicating whether the judgment standard is satisfied. Alternatively, the prompt interpretation degree judgment result may include information indicating the portion of the prompt (e.g., "Akita dog") whose interpretation degree is lower than a predetermined standard. The prompt interpretation degree judgment result may include, for example, information indicating the portion of the prompt (e.g., "Akita dog") that does not satisfy the judgment standard set by the prompt communication device 100A. The prompt interpretation degree judgment result may also include "Akita dog," which did not satisfy the judgment standard, and information indicating its interpretation degree (e.g., 0%). Alternatively, the prompt interpretation degree determination result may include "Akita dog," which did not satisfy the determination criterion, and the ratio of the interpretation degree to the determination criterion (e.g., 0 / 100). In this way, a prompt interpretation degree determination result indicating the prompt portion "Akita dog" with a low interpretation degree allows the prompt communication device 100A to understand from the determination result that the prompt communication device 100B was unable to interpret "Akita dog." Furthermore, if the prompt interpretation degree determination result includes the interpretation degree (or the ratio between the interpretation degree and the determination criterion) in prompt communication device B, the explanation added to the prompt may be adjusted in accordance with the interpretation degree (or the ratio) when modifying the prompt.

[0103] The prompt communication device 100A modifies the image generation prompt A based on the prompt interpretation level determination result. For example, as shown in Figure 13D, the image generation prompt is modified to further describe the characteristics of the "Akita dog." The modified image generation prompt D is then sent again to the prompt communication device 100B.

[0104] The prompt / image conversion unit 1103B of the prompt communication device 100B converts the received image generation prompt D into image data. The converted image data may look like that shown in Figure 13(E), for example. That is, image data that more closely resembles the image data of Figure 13B is generated compared to the image data of Figure 13C.

[0105] In this manner, in this embodiment, prompt communication is repeated between prompt communication devices A and B while modifying the image generation prompts to improve the degree of interpretation. In this manner, the image intended by prompt communication device 100A can be generated by prompt communication device 100B.

[0106] The prompt communication device 100B may update the prompt / image conversion model using the image generation prompt modified by the prompt communication device 100A or the image data sent by the prompt communication device 100A. As a result, the prompt communication device 100B will be able to interpret "Akita dog."

[0107] As described above, in this embodiment, image generation prompts are communicated between devices, and one device modifies the image generation prompt based on the other device's level of prompt interpretation for the prompt so as to improve the other device's level of interpretation. For example, the device modifying the prompt modifies the image generation prompt by adding a description to a portion of the explanation with a low level of interpretation. This allows the receiving device to generate an image that closely resembles the image intended by the prompt provider.

[0108] In the above embodiment, an example of a prompt communication system has been described in which image generation prompts are transmitted and received between the prompt communication device 100A and the prompt communication device 100B. However, prompt communication is not limited to the above example. That is, the counterpart of prompt communication between the prompt communication device 100A and the prompt communication device 100B may be a prompt communication device other than the prompt communication device 100A. Conversely, the counterpart of prompt communication between the prompt communication device 100B and the prompt communication device 100A may be a prompt communication device other than the prompt communication device 100A. For example, an external server may be provided that provides (shares) image generation prompts to multiple prompt communication devices. In this case, the prompt communication device 100A and the prompt communication device 100B may obtain image generation prompts from the external server. Then, the prompt communication device 100A may modify the image generation prompt based on information about the degree of interpretation of the prompt by the prompt communication device 100B so as to improve the degree of interpretation by the prompt communication device 100B. This also makes it possible to generate an image that is closer to the image intended by the prompt provider.

[0109] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0110] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0111] This application claims priority based on Japanese Patent Application No. 2024-047090, filed March 22, 2024, the entire contents of which are incorporated herein by reference.

[0112] 100A...prompt communication device, 100B...prompt communication device, 1101A...prompt acquisition unit, 1102A...prompt correction unit, 1103A...communication unit, 1104A...prompt / image conversion unit, 1101B...communication unit, 1102B...prompt interpretation degree determination unit, 1103B...prompt / image conversion unit, 1104B...image data display unit

Claims

1. An image processing device comprising: an acquisition means for acquiring, from a specified external device, information about the degree of interpretation by the specified external device of a prompt including an explanation about an image to be generated by the generative model; a modification means for modifying the prompt based on the information about the degree of interpretation; and a transmission means for transmitting the modified prompt to the specified external device via a communication means, wherein the modification means modifies the explanation of the prompt so as to increase the amount of the explanation reflected in the image generated by the generative model.

2. An image processing device according to claim 1, characterized in that the information on the degree of interpretation includes information indicating a part of the description where the degree to which the description is reflected in the image does not meet a predetermined standard.

3. The image processing device according to claim 2, wherein said correcting means corrects said prompt by adding a description for a part of said explanation that does not satisfy said predetermined criteria.

4. The image processing device according to claim 2, wherein said correcting means corrects said prompt by replacing a part of said explanation that does not satisfy said predetermined criteria with another explanation.

5. An image processing device as described in claim 2, further comprising: a generation means for inputting the prompt into a generative model to generate an image; and a control means for controlling the generation means to generate an image from the prompt and to transmit the generated image to the specified external device when a specific condition is met.

6. The image processing device described in claim 5, characterized in that the specific condition is satisfied at least when a request to send an image corresponding to the prompt is received from the specified external device, or when the number of parts of the description whose degree of reflection in the image does not satisfy the specified standard exceeds a specified number.

7. The image processing device of claim 5, characterized in that the image processing device repeatedly performs operations including obtaining information about the degree of interpretation, modifying the prompt, and sending the modified prompt, and the specific condition is satisfied if, when the operations are repeated, a part of the description that does not satisfy the specified criterion exists more than a specified number of times.

8. The image processing device according to claim 1, wherein the degree of interpretation of the prompt is determined in units of at least one of words, phrases, clauses, and sentences that make up the explanation.

9. The image processing device according to claim 1, characterized in that the prompt further includes criteria for determining the degree of interpretation of the prompt, and the image processing device further has a setting means for setting the criteria in units of at least one of words, phrases, clauses, and sentences that make up the explanation.

10. The image processing device according to claim 9, wherein said setting means changes said criteria in accordance with a communication history of said prompt with said predetermined external device.

11. An image processing device comprising: a determination means for determining the degree of interpretation for a prompt including an explanation about an image to be generated by a generative model; and a control means for controlling the provision of information about the degree of interpretation to a specified external device via a communication means, wherein after providing the information about the degree of interpretation, the control means obtains from the specified external device a prompt in which the explanation has been modified so that the explanation reflected in the image generated by the generative model is increased.

12. The image processing apparatus according to claim 11, further comprising a generating means for inputting the prompt with the modified explanation into a generative model to generate an image.

13. An image processing device according to claim 11, characterized in that the information on the degree of interpretation includes information indicating a part of the description where the degree to which the description is reflected in the image does not satisfy a predetermined standard.

14. The image processing device of claim 13, wherein the prompt with the modified explanation is modified by adding a description of the portion of the explanation that does not meet the predetermined criteria.

15. The image processing device of claim 13, wherein the prompt with the modified explanation is modified by replacing a portion of the explanation that does not meet the predetermined criteria with another explanation.

16. The image processing device according to claim 13, wherein said control means controls the image generated from said prompt to be acquired from said predetermined external device in response to a specific condition being satisfied.

17. The image processing device described in claim 16, characterized in that the specific condition is satisfied when at least one of the following occurs: an instruction is received from the specified external device requesting an image corresponding to the prompt; or the number of parts of the description whose degree of reflection in the image does not satisfy the specified standard exceeds a specified number.

18. The image processing device of claim 16, characterized in that the image processing device repeatedly performs an operation including providing information about the degree of interpretation and obtaining a modified prompt, and the specific condition is met if, when the operation is repeated, a portion of the explanation that does not satisfy the specified criterion exists more than a specified number of times.

19. The image processing device according to claim 11, wherein the degree of interpretation of the prompt is determined in units of at least one of words, phrases, clauses, and sentences that make up the explanation.

20. The image processing device described in claim 11, characterized in that the prompt further includes criteria for determining the degree of interpretation of the prompt, and the criteria are set by the specified external device in units of at least one of words, phrases, clauses, and sentences that make up the explanation.

21. The image processing device according to claim 20, wherein the criteria are changed according to a history of communication between the prompt and the predetermined external device.

22. A system including a first image processing device and a second image processing device, wherein the first image processing device has: a determination means for determining a degree of interpretation for a prompt including an explanation about an image to be generated by a generative model; and a control means for controlling the second image processing device to provide information about the degree of interpretation to the second image processing device via a communication means; and the second image processing device has: an acquisition means for acquiring information about the degree of interpretation from the first image processing device; a modification means for modifying the prompt based on the information about the degree of interpretation; and a transmission means for transmitting the modified prompt to the first image processing device via a communication means, wherein the modification means modifies the explanation of the prompt so that more of the explanation is reflected in the image generated by the generative model.

23. An image processing method executed in an image processing device, comprising: a step of acquiring, from a specified external device, information regarding the degree of interpretation by the specified external device of a prompt including an explanation about an image to be generated by a generative model; a step of modifying the prompt based on the information about the degree of interpretation; and a step of transmitting the modified prompt to the specified external device via a communication means, wherein the modification step modifies the explanation of the prompt so as to increase the amount of the explanation reflected in the image generated by the generative model.

24. An image processing method executed in an image processing device, comprising: a step of determining the degree of interpretation for a prompt including an explanation about an image to be generated by a generative model; and a step of controlling the provision of information about the degree of interpretation to a specified external device via a communication means, wherein the control step, after providing the information about the degree of interpretation, obtains from the specified external device a prompt in which the explanation has been modified so that the explanation reflected in the image generated by the generative model is increased.

25. A program for causing a computer to function as each means of the image processing device according to any one of claims 1 to 21.

Images