Image acquisition device, prompt learning device, image acquisition method, prompt learning method, and program

Abstract

To provide an image acquisition device capable of reducing the possibility that an undesirable image is output in response to input of a prompt.SOLUTION: In the image acquisition device, in a case where a class of an input image used for learning of a prompt is a suppression target class which is a class for which output of an image is to be suppressed, the processing unit 180 indicates a worse evaluation as similarity between an image feature vector which is a feature vector of the input image and a prompt feature vector which is a feature vector of a combined prompt obtained by combining a base prompt indicating a class in image classification and the class of the input image with a control prompt which is data to be updated is higher. When the class of the input image is a class other than the suppression target class, an evaluation function indicating a better evaluation as the similarity between the image feature vector and the prompt feature vector is higher is used to learn the prompt so that the evaluation indicated by the evaluation function becomes better, and the image is acquired using the learned prompt.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image acquisition device, a prompt learning device, an image acquisition method, a prompt learning method, and a program. [Background technology]

[0002] There are cases where an image is output in response to a prompt input, such as when an image is generated in response to input of text data or the like. For example, the information processing system described in Patent Document 1 inputs a prompt to output information related to desired information input by a user into a language model, and generates text data related to the desired information. This information processing system also inputs a prompt to output an image corresponding to the desired information or the text data related to the desired information into an image generation model, and generates an image related to a topic. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7404596 Summary of the Invention [Problem to be solved by the invention]

[0004] When an image is output in response to a prompt input, it is preferable to be able to reduce the possibility of an undesirable image being output.

[0005] An example of an object of the present disclosure is to provide an image acquisition device, a prompt learning device, an image acquisition method, a prompt learning method, and a program that can solve the above-mentioned problems. [Means for solving the problem]

[0006] According to a first aspect of the present disclosure, an image acquisition device includes: an image feature extraction means for extracting an image feature vector, which is a feature vector of an input image, which is an image of one of the classes in an image classification; a prompt feature extraction means for extracting a prompt feature vector, which is a feature vector of a combined prompt, which is data obtained by combining a base prompt, which is data indicating the class in the image classification and the input image class, which is the class of the input image, and a control prompt, which is data to be updated; a similarity calculation means for calculating the similarity between the prompt feature vector and the image feature vector; a control prompt update means for updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value in which, if the input image class is a suppression target class, which is a class in which image output should be suppressed, the higher the similarity indicated, the worse the evaluation; and, if the input image class is a class other than the suppression target class, the higher the similarity indicated, the better the evaluation; and an output image acquisition means for acquiring an image using the prompt feature vector from the updated control prompt.

[0007] According to a second aspect of the present disclosure, a prompt learning device includes an image feature extraction means for extracting an image feature vector, which is a feature vector of an input image, which is an image of one of the classes in an image classification; a prompt feature extraction means for extracting a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating the class in the image classification and the input image class, which is the class of the input image, and a control prompt, which is data to be updated; a similarity calculation means for calculating the similarity between the prompt feature vector and the image feature vector; and a control prompt update means for updating the value of the control prompt so as to improve the evaluation indicated by the evaluation value using an evaluation function that outputs an evaluation value that indicates a worse evaluation when the input image class is a suppression target class, which is a class for which image output should be suppressed, and indicates a better evaluation when the input image class is a class other than the suppression target class, the higher the similarity indicated by the similarity.

[0008] According to a third aspect of the present disclosure, an image acquisition method includes a computer extracting an image feature vector that is a feature vector of an input image, which is an image of one of classes in an image classification; extracting a prompt feature vector that is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt, which is data to be updated; calculating a similarity between the prompt feature vector and the image feature vector; updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation when the input image class is a suppression target class that is a class for which image output should be suppressed, and indicates a better evaluation when the input image class is a class other than the suppression target class, and acquiring an image using the prompt feature vector based on the updated control prompt.

[0009] According to a fourth aspect of the present disclosure, a prompt learning method includes a computer extracting an image feature vector that is a feature vector of an input image, which is an image of one of classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating the class in image classification and the input image class that is the class of the input image, and a control prompt, which is data to be updated; calculating a similarity between the prompt feature vector and the image feature vector; and updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation when the input image class is a suppression target class, which is a class that should be suppressed from outputting images, and indicates a better evaluation when the input image class is a class other than the suppression target class, the higher the similarity indicated by the similarity.

[0010] According to a fifth aspect of the present disclosure, a program causes a computer to execute the following steps: extracting an image feature vector, which is a feature vector of an input image, which is an image of one of the classes in image classification; extracting a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating the class in image classification and the input image class, which is the class of the input image, and a control prompt, which is data to be updated; calculating the similarity between the prompt feature vector and the image feature vector; updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value where, if the input image class is a suppression target class, which is a class for which image output should be suppressed, the higher the similarity indicated by the similarity, the worse the evaluation; and, if the input image class is a class other than the suppression target class, the higher the similarity indicated by the similarity, the better the evaluation; and acquiring an image using the prompt feature vector from the updated control prompt.

[0011] According to a sixth aspect of the present disclosure, a program causes a computer to execute the following steps: extract an image feature vector, which is a feature vector of an input image, which is an image of one of the classes in image classification; extract a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating the class in image classification and the input image class, which is the class of the input image, and a control prompt, which is data to be updated; calculate a similarity between the prompt feature vector and the image feature vector; and update the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation when the input image class is a suppression target class, which is a class for which image output should be suppressed, and indicates a better evaluation when the input image class is a class other than the suppression target class, the higher the similarity indicated by the similarity. [Effects of the Invention]

[0012] According to one aspect of the present disclosure, when an image is output in response to a prompt input, the possibility of an undesirable image being output can be reduced. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 illustrates an example of the configuration of an image acquisition device according to at least one embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of input and output of data in each part of a processing unit according to at least one embodiment. [Figure 3] 2 is a diagram illustrating a first example of a configuration of an output image acquisition unit and input / output of data according to at least one embodiment. FIG. [Figure 4] FIG. 10 is a diagram illustrating a second example of the configuration of an output image acquisition unit and input / output of data according to at least one embodiment. [Figure 5] FIG. 10 is a diagram illustrating a third example of the configuration of an output image acquisition unit and data input / output according to at least one embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a procedure of a process performed by an image acquisition device according to at least one embodiment. [Figure 7] FIG. 1 illustrates an example configuration of an image acquisition system according to at least one embodiment. [Figure 8] FIG. 1 illustrates an example configuration of a prompt learning device according to at least one embodiment. [Figure 9] FIG. 1 illustrates an example of the configuration of an image acquisition device according to at least one embodiment. [Figure 10] FIG. 1 illustrates an example configuration of a prompt learning device according to at least one embodiment. [Figure 11] 1A to 1C are diagrams illustrating an example of a processing procedure in an image acquisition method according to at least one embodiment. [Figure 12] FIG. 1 illustrates an example of a process flow in a prompt learning method according to at least one embodiment. [Figure 13]FIG. 1 illustrates an example configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0015] First Embodiment 1 is a diagram illustrating an example of the configuration of an image acquisition device according to at least one embodiment. In the configuration illustrated in FIG. 1, image acquisition device 100 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 170, and a processing unit 180. The processing unit 180 includes an input image acquisition unit 181, a base prompt acquisition unit 182, a control prompt setting unit 183, an image feature extraction unit 184, a prompt feature extraction unit 185, a similarity calculation unit 186, a class output unit 187, a loss calculation unit 188, a control prompt update unit 189, an output image acquisition unit 190, and an image output unit 191.

[0016] The image capture device 100 receives prompt input and outputs an image, and in particular, the image capture device 100 reduces the likelihood of outputting an image that is deemed undesirable. The image acquisition device 100 may be configured using a computer such as a personal computer (PC) or a workstation (WS).

[0017] To reduce the possibility of outputting an undesirable image, the image capture device 100 provides an updatable portion of the prompt and updates the prompt using sample data. The prompt here refers to input data for requesting an operation from the device. The prompt may be, but is not limited to, a character string (text data). For example, the prompt, or a portion thereof, may be numerical data. The updateable portion of the prompt is also called a control prompt. The portion of the prompt other than the control prompt is also called a base prompt. The combined control prompt and base prompt (the entire prompt) is also called a combined prompt.

[0018] The combination of prompts here may be the concatenation of prompts as character strings or bit strings. The concatenation of two pieces of data here means joining the end of one piece of data to the beginning of the other piece of data to combine them into one piece of data. However, the method by which the image capture device 100 combines the base prompt and the control prompt is not limited to a specific method. As a method by which the image capture device 100 combines the base prompt and the control prompt, various methods that can decompose the combined prompt into parts (tokens) can be used.

[0019] Updating the prompts is also referred to as learning or training the prompts. Sample data used to learn the prompts is also referred to as training data. Image capture device 100 may learn the prompts using a known machine learning method, such as backpropagation.

[0020] The image capture device 100 uses an image classification technique to train prompts to reduce the likelihood of outputting images of an undesirable class. Now, consider a case where image capture device 100 generates and outputs an image based on a prompt. In the case of image generation, there are a variety of images that can be generated, and the process of reducing the possibility of generating a specific image (e.g., an image that satisfies a specified condition) is considered to be complex.

[0021] In contrast, the image acquisition device 100 uses an image classification technique to reduce the possibility of outputting an image of an undesirable class. With the image acquisition device 100, the number of classes to be classified is relatively small (for example, smaller than the number of images that may be generated in image generation), and it is expected that the possibility of outputting an undesirable image can be reduced with relatively simple processing (relatively simple learning). In image classification, a class (the entire class into which images are classified) is also called an input image class. Among the input image classes, classes that are deemed undesirable are also called suppression target classes. A suppression target class can be considered as a class whose image output should be suppressed.

[0022] Another possible method for reducing the possibility that the image capture device 100 outputs an undesirable image is to retrain the image capture process, such as the image generation process, that the image capture device 100 uses to capture an image. However, the learning cost (training cost) of a process that acquires images based on prompts, such as the image generation process, is likely to be high. For example, learning a process that acquires images based on prompts, such as the image generation process, requires a large amount of training data, and the training process may take a long time.

[0023] In contrast, the learning performed by the image acquisition device 100 can be understood as using the trained machine learning model as is for image acquisition, and fine-tuning the machine learning model that generates input data for the machine learning model for image acquisition so as to reduce the possibility of outputting images that are considered undesirable. Image acquisition device 100 is expected to reduce the possibility of outputting undesirable images with relatively simple processing (relatively simple learning), since it uses a trained machine learning model for image acquisition as is.

[0024] The operator who causes image capture device 100 to learn the prompts may be the same person as the user who requests images from image capture device 100, or may be a different person. For example, when an administrator of the image capture device 100 makes the image capture device 100 available to the public, the administrator may have the image capture device 100 learn prompts to reduce the possibility of the image capture device 100 outputting images that are considered socially undesirable.

[0025] Alternatively, a method for learning prompts may be made public to users of image capture device 100. Then, image capture device 100 may be made to learn prompts in order to reduce the possibility that an image undesirable to the user (an image not desired by the user) will be output when the user requests an image from image capture device 100. When multiple users share one image capture device 100, image capture device 100 may store learned prompts (prompts obtained through learning) for each user.

[0026] The communication unit 110 communicates with other devices. For example, the communication unit 110 may receive image data used as an input image (image data used as part of training data) from other devices. Display unit 120 has a display screen such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, display unit 120 may store prompts and sample data for learning the prompts.

[0027] The operation input unit 130 includes input devices such as a keyboard and a mouse, and accepts user operations. For example, the operation input unit 130 may accept input operations for settings related to prompt learning, such as a learning rate for prompt learning. The operation input unit 130 may also accept input operations for base prompts.

[0028] The storage unit 170 stores various data. For example, the storage unit 170 may store training data, base prompts, control prompts, combined prompts, evaluation functions for learning prompts, and settings for learning prompts, such as a learning rate, or some of these. The storage unit 170 is configured using a storage device provided in the image acquisition device 100.

[0029] The processing unit 180 performs various processes by controlling each unit of the image acquisition device 100. The functions of the processing unit 180 are performed, for example, by a CPU (Central Processing Unit) included in the image acquisition device 100 reading and executing a program from the storage unit 170.

[0030] FIG. 2 is a diagram showing an example of input and output of data at each part of the processing unit 180. As shown in FIG. The input image acquisition unit 181 acquires one or more images including an image of a class to be suppressed. The image acquired by the input image acquisition unit 181 is also referred to as an input image. A combination of an input image and a prompt indicating the class of the input image is used as training data for the image acquisition device 100 to learn the prompt. The class of the image here refers to the class into which the image is classified by class classification. The input image acquisition unit 181 outputs the input image to the image feature extraction unit 184 .

[0031] The method by which the input image acquisition unit 181 acquires the input image is not limited to a specific method. For example, the input image acquisition unit 181 may acquire training data prepared in another device. Alternatively, the input image acquisition unit 181 may acquire the input image from another device in accordance with a user operation.

[0032] Alternatively, the input image acquisition unit 181 may receive a keyword indicating an input image class and search for the input image using the specified keyword. For example, the input image may be searched for by an internet search using the specified keyword. Alternatively, the input image acquisition unit 181 may receive a prompt including a keyword and search for the input image using a base model that outputs an image corresponding to the keyword.

[0033] Since input image acquisition unit 181 searches for an input image, the operator who causes image acquisition device 100 to learn a prompt does not need to input an input image to image acquisition device 100. In this respect, image acquisition device 100 is expected to reduce the burden on the operator who causes image acquisition device 100 to learn a prompt. The keyword may be specified for the input image acquisition unit 181 by inputting a base prompt including the keyword into the image acquisition device 100 .

[0034] The base prompt acquisition unit 182 acquires a base prompt. In particular, the base prompt acquisition unit 182 acquires, for each input image, a base prompt that indicates the class of the input image (input image class). The base prompt acquisition unit 182 outputs the base prompt to the prompt feature extraction unit 185. In particular, the base prompt acquisition unit 182 outputs to the prompt feature extraction unit 185 a base prompt indicating a class in the image classification and an input image class.

[0035] The base prompt is used as a correct label (teaching data) for the class of the input image when the image capture device 100 learns the prompt, and is also used as a prompt indicating a request for the output image when the image capture device 100 captures the output image. The output image here refers to an image acquired and output by the image acquisition device 100. The image acquisition device 100 may generate the output image. Alternatively, the image acquisition device 100 may acquire the output image by performing an image search.

[0036] The same base prompt may be used when learning the prompt and when acquiring the output image, or different base prompts may be used. If a different base prompt is used when acquiring the output image than when learning the prompt, image acquisition device 100 may combine the base prompt for acquiring the output image with the learned control prompt to use as the prompt for acquiring the output image.

[0037] The base prompt may include data indicating all or some of the classes in the image classification in addition to the input image class. For example, the base prompt may include keywords for each class in the image classification and keywords for the output image class. In this case, image capture device 100 may distinguish between the keywords for each class in the image classification and the keywords for the output image class based on the position of the keywords in the base prompt.

[0038] However, the method by which the image acquisition device 100 acquires information indicating a class in image classification (information identifying a class), such as a keyword for the class in image classification, is not limited to a specific method. For example, the storage unit 170 may store keywords for each class in image classification in advance. Alternatively, the base prompt acquisition unit 182 may tally the input image classes indicated in the base prompts for all base prompts and generate a set of input image classes. Here, "set" means that the classes are tallyed so that the same class is not duplicated. Then, the image acquisition device 100 may use the set of input image classes as all classes in image classification.

[0039] If the base prompt acquired by the base prompt acquisition unit 182 does not include data indicating all classes in the image classification, the data indicating all classes in the image classification may be inserted into the base prompt and output to the prompt feature extraction unit 185. When the base prompt acquisition unit 182 inserts data indicating a class in image classification into a base prompt indicating an input image class, this can be considered as acquiring a base prompt indicating a class in image classification and an input image class.

[0040] In addition to the input image class, the base prompt may also include data indicating a suppression target class. For example, each base prompt may include a flag indicating whether the input image class indicated by the base prompt corresponds to a suppression target class. However, the method by which the image acquisition device 100 acquires information indicating the suppression target class is not limited to a specific method. For example, the storage unit 170 may store keywords of the suppression target class in advance. If the base prompt acquired by the base prompt acquisition unit 182 includes data indicating a class to be suppressed, the data indicating the class to be suppressed may be deleted from the base prompt and output to the prompt feature extraction unit 185.

[0041] The control prompt setting unit 183 sets a control prompt. That is, the control prompt setting unit 183 sets a value of the control prompt. The control prompt setting unit 183 may set a random value as the initial value of the control prompt. For example, the control prompt setting unit 183 sets a random value as the initial value of the control prompt. A random character string of text may be set. Furthermore, when the control prompt update unit 189 updates the control prompt in prompt learning, the control prompt setting unit 183 may use the updated control prompt as is. The control prompt setting unit 183 outputs the control prompt to the prompt feature extraction unit 185 .

[0042] The image feature extraction unit 184 extracts a feature vector of each input image, which is also referred to as an image feature vector. The image feature extraction unit 184 is an example of an image feature extraction means. The image feature extraction unit 184 outputs the image feature vector to the similarity calculation unit 186 .

[0043] The prompt feature extraction unit 185 generates a combined prompt by combining a base prompt and a control prompt, and breaks down the resulting combined prompt into elements (tokens). The prompt feature extraction unit 185 then extracts a feature vector for each of the resulting elements. The feature vector for each element generated by the prompt feature extraction unit 185 is also referred to as a prompt feature vector. The prompt feature vector can be considered as the feature vector of the combined prompt. The prompt feature extraction unit 185 is an example of a prompt feature extraction means.

[0044] The prompt feature extraction unit 185 outputs the prompt feature vector to the similarity calculation unit 186 . Furthermore, if the same base prompt is used when learning the prompt and when acquiring the output image, the prompt feature extraction unit 185 outputs the prompt feature vector of the learned combined prompt to the output image acquisition unit 190. The learned combined prompt here is a prompt obtained by combining the base prompt acquired by the base prompt acquisition unit 182 when learning the prompt and the learned control prompt.

[0045] On the other hand, if different base prompts are used when learning the prompt and when acquiring the output image, the prompt feature extraction unit 185 outputs to the output image acquisition unit 190 a prompt feature vector of a combined prompt that combines the base prompt for acquiring the output image and the learned control prompt.

[0046] The prompt feature extraction unit 185 outputs the prompt feature vector to the similarity calculation unit 186 and the output image acquisition unit 190, which can be considered as using the same text encoder to classify images when learning the prompt and to acquire the output image (for example, generate the output image).

[0047] The similarity calculation unit 186 calculates the similarity between the image feature vector extracted by the image feature extraction unit 184 and the prompt feature vector extracted by the prompt feature extraction unit 185 for each combination of an input image and a base prompt in the training data. In particular, the similarity calculation unit 186 calculates the similarity between the feature vector of each element of the combined prompt and the image feature vector of that element. The similarity for each element of the combined prompt calculated by the similarity calculation unit 186 can be used as data indicating the likelihood that input data will be classified into that class for each class. The similarity calculation unit 186 is an example of a similarity calculation means.

[0048] The similarity calculation unit 186 outputs the calculated similarity to the class output unit 187 and the loss calculation unit 188. In the following, an example will be described in which the similarity calculation unit 186 uses cosine similarity as the similarity, but the similarity used by the similarity calculation unit 186 is not limited to a specific one, and various similarities that can calculate the similarity between two vectors can be used.

[0049] The similarity calculation unit 186 may use a similarity in which the larger the index value of the similarity, the higher the similarity between the two vectors. For example, when the similarity calculation unit 186 uses a cosine similarity, the larger the value of the cosine similarity, the higher the similarity between the two vectors.

[0050] Alternatively, the similarity calculation unit 186 may use a similarity in which the smaller the similarity index value, the higher the similarity between two vectors. For example, the similarity calculation unit 186 may use the Euclidean distance in vector space as the similarity. In this case, the smaller the Euclidean distance value, the higher the similarity between two vectors.

[0051] The class output unit 187 detects the class with the highest similarity calculated by the similarity calculation unit 186. The class detected by the class output unit 187 can be considered as the class with the highest (largest) likelihood that the input image will be classified into that class. The class with the highest likelihood that the input image will be classified into that class is also called the most likely class. The class output unit 187 outputs information indicating the detected class to the loss calculation unit 188. Outputting information indicating the class is also referred to as outputting the class.

[0052] The loss calculation unit 188 calculates the value of the loss function used in learning the prompt. The loss calculation unit 188 outputs the calculated loss value (value of the loss function) to the control prompt update unit 189.

[0053] However, the evaluation function used by the image capture device 100 for learning prompts is not limited to a loss function. The image capture device 100 may use an evaluation function value in which a larger evaluation value indicates a better evaluation. When the image capture device 100 uses a loss function as the evaluation function, a smaller loss value can be considered to indicate a better evaluation.

[0054] In addition, the image acquisition device 100 uses an evaluation function that, for classes to be suppressed, the more similar the input image vector and the prompt feature vector are, the worse the evaluation, and for classes other than the classes to be suppressed, the more similar the input image vector and the prompt feature vector are, the better the evaluation.

[0055] Specifically, when the input image class is a suppression target class, the evaluation function used by the image acquisition device 100 outputs an evaluation value that indicates a worse evaluation as the similarity between the image feature vector and the prompt feature vector increases (the similarity increases). For example, when the image acquisition device 100 uses a loss function, when the input image class is a suppression target class, the loss value increases as the similarity between the image feature vector and the prompt feature vector increases.

[0056] Furthermore, when the input image class is a class other than the suppression target class, the evaluation function used by the image acquisition device 100 outputs an evaluation value that indicates a better evaluation the greater the similarity (the higher the similarity) between the image feature vector and the prompt feature vector. For example, when the image acquisition device 100 uses a loss function, when the input image class is a class other than the suppression target class, the greater the similarity between the image feature vector and the prompt feature vector, the smaller the loss value.

[0057] In calculating the evaluation function value, the loss calculation unit 188 uses the feature vector of the element corresponding to the input image class from among the feature vectors extracted for each element of the combined prompt by the similarity calculation unit 186. Therefore, in calculating the evaluation function value, the loss calculation unit 188 uses the likelihood that the input image will be classified into the input image class as the similarity between the image feature vector and the prompt feature vector. The loss calculation unit 188 may use the input image class indicated in the base prompt as the input image class, or may use the most likely class detected by the class output unit 187 as the input image class.

[0058] The control prompt update unit 189 updates the control prompts so as to reduce the loss value calculated by the loss calculation unit 188. The update of the control prompts performed by the control prompt update unit 189 can be considered as learning the prompts so that the classification accuracy of images in the suppression target class decreases and the classification accuracy of images in classes other than the suppression class increases. The control prompt update unit 189 is an example of a control prompt update means. The control prompt update unit 189 outputs the updated control prompt to the control prompt setting unit 183 .

[0059] As described above with respect to the image acquisition device 100, the control prompt update unit 189 may update the control prompt using a machine learning method that uses the derivative of a loss function, such as backpropagation. Here, the combination of the prompt feature extraction unit 185, the similarity calculation unit 186, and the loss calculation unit 188 can be considered to correspond to a loss function that takes the control prompt as an argument. The loss function resulting from the combination of the functions of these units may be given as a mathematical formula, and the prompt feature extraction unit 185 may update the control prompt using the derivative of the loss function expressed by the mathematical formula.

[0060] When the control prompt update unit 189 converts the feature vector of the control prompt (the feature vector for each element of the control prompt) into a character string (text), the storage unit 170 may store a data table in which, for each character string that can be used as an element value of the control prompt, it is associated with the feature vector (value of the feature vector) into which the prompt feature extraction unit 185 converts the character string. Then, the control prompt update unit 189 may refer to the data table to convert the feature vector of the control prompt into a character string.

[0061] The output image acquisition unit 190 acquires an output image using a prompt feature vector based on the learned control prompt. The output image acquisition unit 190 may generate the output image. Alternatively, the output image acquisition unit 190 may acquire the output image by image search. The output image acquisition unit 190 corresponds to an example of an output image acquisition means. The output image acquisition unit 190 outputs the output image to the image output unit 191.

[0062] The image output unit 191 outputs an output image. The method by which the image output unit 191 outputs the output image is not limited to a specific method. For example, the image output unit 191 may control the display unit 120 to display the output image. Alternatively, the image output unit 191 may control the communication unit 110 to transmit the output image to another device.

[0063] In the example of Figure 2, the processing performed by the combination of input image acquisition unit 181, base prompt acquisition unit 182, control prompt setting unit 183, image feature extraction unit 184, prompt feature extraction unit 185, similarity calculation unit 186, class output unit 187, loss calculation unit 188, and control prompt update unit 189 can be considered as prompt learning using an image classification task. Furthermore, the processing performed by the combination of base prompt acquisition unit 182, control prompt setting unit 183, prompt feature extraction unit 185, output image acquisition unit 190, and image output unit 191 can be considered as an output image acquisition task.

[0064] The image capture device 100 can be thought of as sharing the control prompts and the text encoder function of the prompt feature extractor 185 between the image classification task and the output image capture task. As a result, it is expected that the classification accuracy of images in the suppression target class will decrease when the output image acquisition task is performed, just as it does when prompt learning is performed using the image classification task, in the image acquisition device 100. This is expected to reduce the possibility that the output image acquisition unit 190 will acquire images classified into the suppression target class.

[0065] When generating an output image, the output image acquisition unit 190 is likely to be less likely to recognize a request for an output image indicated in the base prompt as a request for an image classified as a suppression target class. This is expected to reduce the possibility that the output image acquisition unit 190 will generate an image classified as a suppression target class.

[0066] When the output image acquisition unit 190 acquires an output image through an image search, it is considered that the possibility of recognizing an image that is a candidate in the image search as an image that is classified into a suppression target class is reduced. As a result, even if the request for an output image indicated in the base prompt is a request for an image that is classified into a suppression target class, it is expected that the possibility of the output image acquisition unit 190 recognizing an image that is a candidate in the image search as an image that is classified into a suppression target class and acquiring it is reduced.

[0067] Fig. 3 is a diagram showing a first example of the configuration and data input / output of output image acquisition unit 190. In the example of Fig. 3, output image acquisition unit 190 includes an image generation unit 291. Output image acquisition unit 190 in the example of Fig. 3 is also referred to as output image acquisition unit 190a. The image generation unit 291 receives a prompt feature vector based on the learned control prompt and generates an output image. The image generation unit 291 outputs the generated output image to the image output unit 191. As described above in the case where the output image acquisition unit 190 generates an output image, it is expected that the possibility that the image generation unit 291 generates an image that is classified into a suppression target class is reduced.

[0068] The function of an existing base model that generates an image upon receiving a prompt input may be used as the function of the image generation unit 291. Each unit for learning the prompt may be built into an existing base model system that generates an image upon receiving a prompt input.

[0069] Fig. 4 is a diagram showing a second example of the configuration and data input / output of output image acquisition unit 190. In the example of Fig. 4, output image acquisition unit 190 includes an image generation unit 291 and an image-image search unit 292. Output image acquisition unit 190 in the example of Fig. 4 is also referred to as output image acquisition unit 190b.

[0070] The function of the image generation unit 291 in the example of Fig. 4 is similar to the function of the image generation unit 291 in the example of Fig. 3. In the example of Fig. 4, the image generation unit 291 outputs the generated image to the image-image search unit 292. The image-image search unit 292 performs an image-image search using the image generated by the image generation unit 291 as a search image. The image-image search here refers to searching for an image similar to the search image.

[0071] As described above regarding the case where the output image acquisition unit 190 generates an output image, it is expected that the possibility that the image generation unit 291 will generate an image classified into the suppression target class will be reduced. This will reduce the possibility that the image-image search unit 292 will perform an image-image search using an image classified into the suppression target class as a search image, and it is expected that the possibility that the image-image search unit 292 will acquire an image classified into the suppression target class as a search result will be reduced.

[0072] As in the case of the example of Fig. 3, in the case of the example of Fig. 4, the function of an existing base model that receives a prompt input and generates an image may be used as the function of the image generation unit 291. Each unit for learning the prompt may be built into an existing base model system that receives a prompt input and generates an image. Furthermore, the function of an existing searcher that performs an image-image search may be used as the function of the image-image search unit 292. The image-image search unit 292 may be configured using an existing searcher that performs an image-image search.

[0073] Fig. 5 is a diagram showing a third example of the configuration and data input / output of output image acquisition unit 190. In the example of Fig. 5, output image acquisition unit 190 includes a text-image search unit 293. Output image acquisition unit 190 in the example of Fig. 5 is also referred to as output image acquisition unit 190c. The text-image search unit 293 performs an image search using the prompt feature vector. The image search performed by the text-image search unit 293 can be considered as a text-image search using a combined prompt. The text-image search here refers to a search for an image that has a feature indicated by a character string (text data). As described above when the output image acquisition unit 190 acquires an output image through image search, it is expected that the possibility that the text-image search unit 293 will acquire an image classified into a class to be suppressed through image search will be reduced.

[0074] The function of an existing base model that searches for images in response to a prompt input may be used as the function of the text-image search unit 293. Each unit for learning prompts may be built into an existing base model system that searches for images in response to a prompt input.

[0075] FIG. 6 is a diagram showing an example of the procedure of processing performed by the image acquisition device 100. 6, the control prompt setting unit 183 initializes a control prompt (step S101). For example, the control prompt setting unit 183 sets the initial value of the control prompt to a random value.

[0076] Next, image acquisition device 100 acquires training data for learning a prompt (step S102). Specifically, input image acquisition unit 181 acquires an input image, and base prompt acquisition unit 182 acquires a base prompt.

[0077] Next, the control prompt update unit 189 updates the control prompt (step S103). As described above, the control prompt update unit 189 updates the control prompt so that, in the suppression target class, the loss increases as the input image vector and the prompt feature vector are more similar, and in the classes other than the suppression target class, the loss decreases as the input image vector and the prompt feature vector are more similar, so that the value (loss) of the loss function becomes smaller.

[0078] Next, the processing unit 180 determines whether to terminate the learning of the prompt (step S104). Specifically, the processing unit 180 determines whether an end condition for the learning of the prompt is met. The end condition here is not limited to a specific condition. For example, the end condition here may be a condition that the control prompt update unit 189 has repeated updating the control prompt in step S103 a predetermined number of times or more. Alternatively, the end condition here may be a condition that the likelihood of classifying an image of the suppression target class into the suppression target class is equal to or less than a predetermined value, and the likelihood of classifying an image of a class other than the suppression target class into the correct class is equal to or greater than a predetermined value.

[0079] If the processing unit 180 determines that the learning of the prompt should not be ended (step S104: NO), the process returns to step S103. On the other hand, if the processing unit 180 determines that the learning of the prompt is to be terminated (step S104: YES), the output image acquisition unit 190 acquires an output image (step S105). As described above, the output image acquisition unit 190 acquires an output image using the prompt feature vector of the combined prompt that uses the learned control prompt.

[0080] Next, the image output unit 191 outputs the output image (step S106). As described above, the method by which the image output unit 191 outputs the output image is not limited to a specific method. After step S106, the image acquisition device 100 ends the processing of FIG.

[0081] As described above, the image feature extraction unit 184 extracts an image feature vector, which is a feature vector of an input image that is an image of one of the classes in the image classification. The prompt feature extraction unit 185 extracts a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating a class in image classification and an input image class, which is the class of the input image, and a control prompt, which is data to be updated. The similarity calculation unit 186 calculates the similarity between the prompt feature vector and the image feature vector.

[0082] The control prompt update unit 189 updates the value of the control prompt so that the evaluation indicated by the evaluation value becomes better, using an evaluation function that outputs an evaluation value that indicates a worse evaluation the higher the similarity indicated by the similarity when the input image class is a suppression target class, which is a class for which image output should be suppressed, and that indicates a better evaluation the higher the similarity indicated by the similarity when the input image class is a class other than a suppression target class. The output image acquisition unit 190 acquires an image using the prompt feature vector according to the updated control prompt.

[0083] Image acquisition device 100 can reduce the possibility of outputting an image that is deemed undesirable. Specifically, the control prompt update unit 189 uses an evaluation function that outputs an evaluation value that indicates a worse evaluation the higher the similarity indicated when the input image class is a suppression target class, which is a class where image output should be suppressed, and that outputs an evaluation value that indicates a better evaluation the higher the similarity indicated when the input image class is a class other than the suppression target class, and learns the control prompt (updates the value of the control prompt) so that the evaluation indicated by the evaluation value becomes better, which is thought to result in a decrease in the classification accuracy of images of the suppression target class.

[0084] Furthermore, when the output image acquisition unit 190 acquires an output image using the learned control prompt and a prompt feature vector obtained by the same feature extraction method as used when learning the control prompt, it is conceivable that the classification accuracy of images in the suppression target class will be relatively low even when the output image is acquired. Since the classification accuracy of images in the suppression target class is relatively low, it is expected that the output image acquisition unit 190 will be less likely to acquire images in the suppression target class.

[0085] Furthermore, according to the image acquisition device 100, when an image is acquired by the output image acquisition unit 190, a trained machine learning model is used as is, and it is expected that the possibility of outputting an undesirable image can be reduced with relatively simple processing (relatively simple learning). Furthermore, the image acquisition device 100 uses an image classification technique to reduce the possibility of outputting an image of an undesirable class, and is therefore expected to reduce the possibility of outputting an undesirable image with relatively simple processing (relatively simple learning) compared to, for example, reducing the possibility of generating an undesirable image in an image generation process.

[0086] The output image acquisition unit 190 also generates an image using the prompt feature vector based on the updated control prompt. The image acquisition device 100 is expected to reduce the possibility of generating an image that is classified into a suppression target class.

[0087] Furthermore, the output image acquisition unit 190 acquires an image by image search using an image generated using a prompt feature vector based on the updated control prompt. According to the image acquisition device 100, it is expected that the possibility of generating an image classified into a suppression target class is reduced, and that the possibility of acquiring an image classified into a suppression target class as a result of an image search using the generated image is reduced.

[0088] Furthermore, the output image acquisition unit 190 acquires an image through an image search using the prompt feature vector based on the updated control prompt. The image acquisition device 100 is expected to reduce the possibility of acquiring images classified into a suppression target class through image search.

[0089] The input image is an image obtained by an image search using a keyword indicating the input image class. By having image acquisition device 100 search for the input image, the operator who causes image acquisition device 100 to learn the prompt does not need to input the input image into image acquisition device 100. In this respect, image acquisition device 100 is expected to reduce the burden on the operator who causes image acquisition device 100 to learn the prompt.

[0090] Second Embodiment 7 is a diagram illustrating an example of the configuration of an image capture system according to at least one embodiment. In the configuration shown in FIG. 7, image capture system 10 includes prompt learning device 300 and image capture device 400.

[0091] The prompt learning device 300 learns prompts in the same manner as the image capture device 100. The prompt learning device 300 then transmits the learned combined prompts to the image capture device 400. The image capture device 400 captures and outputs an output image using the trained combined prompts received from the prompt training device 300 .

[0092] Alternatively, the prompt learning device 300 may transmit the learned control prompt to the image capture device 400. The image capture device 400 may then acquire a base prompt and combine the acquired base prompt with the learned control prompt to generate a combined prompt. The image capture device 400 may then acquire an image using the generated combined prompt.

[0093] 8 is a diagram showing an example of the configuration of prompt learning device 300. In the configuration shown in FIG. 8, prompt learning device 300 includes communication unit 110, display unit 120, operation input unit 130, storage unit 170, and processing unit 380. The processing unit 380 includes an input image acquisition unit 181, a base prompt acquisition unit 182, a control prompt setting unit 183, an image feature extraction unit 184, a prompt feature extraction unit 185, a similarity calculation unit 186, a class output unit 187, a loss calculation unit 188, a control prompt update unit 189, and a prompt output unit 392.

[0094] 8 that have the same functions as those of the corresponding parts in FIG. 1 are given the same reference numerals (110, 120, 130, 170, 181, 182, 183, 184, 185, 186, 187, 188, 189), and detailed description thereof will be omitted here. Prompt learning device 300 differs from image capture device 100 in that processing unit 380 does not include output image capture unit 190 and image output unit 191, which are included in processing unit 180 of image capture device 100, and instead includes prompt output unit 392. In all other respects, prompt learning device 300 is similar to image capture device 100.

[0095] The prompt output unit 392 transmits the combined prompt learned by the image capture device 100 to the image capture device 400 via the communication unit 110 . Image acquisition system 10 can be considered as an image acquisition device 400 that is a separate device from image acquisition device 100, in which output image acquisition section 190 and image output section 191 of image acquisition device 100 are configured.

[0096] As in the case of output image acquisition unit 190a in Figure 3, image acquisition device 400 may generate an output image. In this case, an existing base model that receives a prompt input and generates an image may be used as image acquisition device 400. Then, prompt feature extraction unit 185 of prompt learning device 300 may extract a prompt feature vector using the same process as the process performed by text encoding of the existing base model.

[0097] As in the case of the output image acquisition unit 190b in FIG. 4, the image acquisition device 400 may generate an image and use the generated image as a search image to search for an output image. In this case, the image acquisition device 400 may be configured to use a combination of an existing foundation model that receives a prompt input and generates an image, and an existing image searcher that performs an image-to-image search. The prompt feature extraction unit 185 of the prompt learning device 300 may then extract a prompt feature vector using the same process as the text encoding process of the existing foundation model. Alternatively, the image-to-image extractor may perform an image search using the image generated by the foundation model as a search image.

[0098] As in the case of output image acquisition unit 190c in Figure 5, image acquisition device 400 may acquire an output image through image search. In this case, image acquisition device 400 may use an existing base model that receives a prompt input and performs an image search. Then, prompt feature extraction unit 185 of prompt learning device 300 may extract a prompt feature vector using the same process as the process performed by text encoding in the existing base model.

[0099] As described above, the image feature extraction unit 184 extracts an image feature vector, which is a feature vector of an input image that is an image of one of the classes in the image classification. The prompt feature extraction unit 185 extracts a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating a class in image classification and an input image class, which is the class of the input image, and a control prompt, which is data to be updated.

[0100] The similarity calculation unit 186 calculates the similarity between the prompt feature vector and the image feature vector. The control prompt update unit 189 updates the value of the control prompt so that the evaluation indicated by the evaluation value becomes better, using an evaluation function that outputs an evaluation value that indicates a worse evaluation the higher the similarity indicated by the similarity when the input image class is a suppression target class, which is a class for which image output should be suppressed, and that indicates a better evaluation the higher the similarity indicated by the similarity when the input image class is a class other than a suppression target class.

[0101] According to the prompt learning device 300, by inputting a learned prompt from the prompt learning device 300 into a device that receives a prompt input and acquires an image, it is expected that when an image is output in response to the input of a prompt, the possibility of an undesirable image being output can be reduced.

[0102] The input image is an image obtained by an image search using a keyword indicating the input image class. Because prompt learning device 300 searches for input images, the operator who has prompt learning device 300 learn prompts does not need to input input images into prompt learning device 300. In this respect, prompt learning device 300 is expected to reduce the burden on the operator who has prompt learning device 300 learn prompts.

[0103] Third Embodiment 9 is a diagram illustrating an example of the configuration of an image capture device according to at least one embodiment. In the configuration illustrated in FIG. 9, an image capture device 610 includes an image feature extraction unit 611, a prompt feature extraction unit 612, a similarity calculation unit 613, a control prompt update unit 614, and an output image capture unit 615.

[0104] With this configuration, the image feature extraction unit 611 extracts an image feature vector, which is a feature vector of an input image that is an image of one of the classes in the image classification. The prompt feature extraction unit 612 extracts a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating the class in the image classification and the input image class, which is the class of the input image, and a control prompt, which is the data to be updated. The similarity calculation unit 613 calculates the similarity between the prompt feature vector and the image feature vector.

[0105] The control prompt update unit 614 updates the value of the control prompt using an evaluation function that outputs an evaluation value that indicates a worse evaluation the higher the similarity indicated by the similarity when the input image class is a suppression target class, which is a class for which image output should be suppressed, and that outputs an evaluation value that indicates a better evaluation the higher the similarity indicated by the similarity when the input image class is a class other than the suppression target class, so that the evaluation indicated by the evaluation value becomes better. The output image acquisition unit 615 acquires an image using the prompt feature vector according to the updated control prompt.

[0106] The image feature extraction unit 611 corresponds to an example of an image feature extraction means. The prompt feature extraction unit 612 corresponds to an example of a prompt feature extraction means. The similarity calculation unit 613 corresponds to an example of a similarity calculation means. The control prompt update unit 614 corresponds to an example of a control prompt update means. The output image acquisition unit 615 corresponds to an example of an output image acquisition means.

[0107] Image capture device 610 can reduce the likelihood of outputting images that are deemed undesirable. Specifically, the control prompt update unit 614 uses an evaluation function that outputs an evaluation value that indicates a worse evaluation the higher the similarity indicated by the similarity when the input image class is a suppression target class, which is a class where image output should be suppressed, and that outputs an evaluation value that indicates a better evaluation the higher the similarity indicated by the similarity when the input image class is a class other than the suppression target class, and learns the control prompt (updates the value of the control prompt) so that the evaluation indicated by the evaluation value becomes better, which is thought to result in a decrease in the classification accuracy of images of the suppression target class.

[0108] Furthermore, since the output image acquisition unit 615 acquires an output image using the learned control prompt and a prompt feature vector obtained by the same feature extraction method as used when learning the control prompt, it is conceivable that the classification accuracy of images in the suppression target class will be relatively low even when the output image is acquired. Since the classification accuracy of images in the suppression target class is relatively low, it is expected that the output image acquisition unit 615 will be less likely to acquire images in the suppression target class.

[0109] Furthermore, according to the image acquisition device 610, the image acquisition by the output image acquisition unit 615 uses a trained machine learning model as is, and it is expected that the possibility of outputting an undesirable image can be reduced with relatively simple processing (relatively simple learning). Furthermore, the image acquisition device 610 uses an image classification technique to reduce the possibility of outputting an image of an undesirable class, and is therefore expected to reduce the possibility of outputting an undesirable image with relatively simple processing (relatively simple learning) compared to, for example, reducing the possibility of generating an undesirable image in an image generation process.

[0110] The image feature extraction unit 611 can be realized, for example, by using the functions of the image feature extraction unit 184 in Fig. 1, etc. The prompt feature extraction unit 612 can be realized, for example, by using the functions of the prompt feature extraction unit 185 in Fig. 1, etc. The similarity calculation unit 613 can be realized, for example, by using the functions of the similarity calculation unit 186 in Fig. 1, etc. The control prompt update unit 614 can be realized, for example, by using the functions of the control prompt update unit 189 in Fig. 1, etc. The output image acquisition unit 615 can be realized, for example, by using the functions of the output image acquisition unit 190 in Fig. 1, etc.

[0111] <Fourth embodiment> 10 is a diagram illustrating an example of the configuration of a prompt learning device according to at least one embodiment. In the configuration illustrated in FIG. 10, a prompt learning device 620 includes an image feature extractor 621, a prompt feature extractor 622, a similarity calculator 623, and a control prompt updater 624.

[0112] With this configuration, the image feature extraction unit 621 extracts an image feature vector, which is a feature vector of an input image that is an image of one of the classes in the image classification. The prompt feature extraction unit 622 extracts a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating a class in image classification and an input image class, which is the class of the input image, and a control prompt, which is data to be updated. The similarity calculation unit 623 calculates the similarity between the prompt feature vector and the image feature vector. The control prompt update unit 624 updates the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation the higher the similarity indicated by the similarity when the input image class is a suppression target class, which is a class for which image output should be suppressed, and that outputs an evaluation value that indicates a better evaluation the higher the similarity indicated by the similarity when the input image class is a class other than a suppression target class.

[0113] The image feature extraction unit 621 corresponds to an example of an image feature extraction means. The prompt feature extraction unit 622 corresponds to an example of a prompt feature extraction means. The similarity calculation unit 623 corresponds to an example of a similarity calculation means. The control prompt update unit 624 corresponds to an example of a control prompt update means.

[0114] According to the prompt learning device 620, by inputting a learned prompt from the prompt learning device 620 into a device that receives a prompt input and acquires an image, it is expected that when an image is output in response to the input of a prompt, the possibility of an undesirable image being output can be reduced.

[0115] The image feature extraction unit 621 can be realized, for example, by using the functions of the image feature extraction unit 184 in Fig. 1, etc. The prompt feature extraction unit 622 can be realized, for example, by using the functions of the prompt feature extraction unit 185 in Fig. 1, etc. The similarity calculation unit 623 can be realized, for example, by using the functions of the similarity calculation unit 186 in Fig. 1, etc. The control prompt update unit 624 can be realized, for example, by using the functions of the control prompt update unit 189 in Fig. 1, etc.

[0116] Fifth Embodiment 11 is a diagram illustrating an example of a processing procedure in an image acquisition method according to at least one embodiment. The image acquisition method illustrated in FIG. 11 includes extracting image features (step S611), extracting prompt features (step S612), calculating similarity (step S613), updating the control prompt (step S614), and acquiring an output image (step S615).

[0117] In extracting image features (step S611), the computer extracts an image feature vector, which is a feature vector of an input image that is an image of one of the classes in the image classification. In extracting the prompt features (step S612), the computer extracts a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating a class in image classification and an input image class, which is the class of the input image, and a control prompt, which is data to be updated. In calculating the similarity (step S613), the computer calculates the similarity between the prompt feature vector and the image feature vector.

[0118] In updating the control prompt (step S614), the computer updates the value of the control prompt so that the evaluation indicated by the evaluation value becomes better, using an evaluation function that outputs an evaluation value that indicates a worse evaluation the higher the similarity indicated by the similarity when the input image class is a suppression target class, which is a class for which image output should be suppressed, and that indicates a better evaluation the higher the similarity indicated by the similarity when the input image class is a class other than a suppression target class. In obtaining an output image (step S615), the computer obtains an image using the prompt feature vector according to the updated control prompt.

[0119] According to the image acquisition method shown in FIG. 11, it is possible to reduce the possibility that an image that is deemed undesirable will be output. Specifically, if the input image class is a suppression target class, which is a class in which image output should be suppressed, the higher the similarity indicated, the worse the evaluation, and if the input image class is a class other than the suppression target class, the higher the similarity indicated, the better the evaluation is output using an evaluation function that outputs an evaluation value that indicates a better evaluation, and by learning the control prompt (updating the value of the control prompt) so that the evaluation indicated by the evaluation value becomes better, it is thought that the classification accuracy of images of the suppression target class will decrease.

[0120] Furthermore, by acquiring an output image using the trained control prompt and a prompt feature vector obtained using the same feature extraction method as when training the control prompt, it is possible that the classification accuracy of images in the suppression target class will be relatively low even when acquiring the output image. Since the classification accuracy of images in the suppression target class is relatively low, it is expected that the possibility of acquiring images in the suppression target class will be relatively low.

[0121] Furthermore, according to the image acquisition method shown in Figure 11, since the trained machine learning model is used as is for image acquisition, it is expected that the possibility of outputting an undesirable image can be reduced with relatively simple processing (relatively simple learning). Furthermore, according to the image acquisition method shown in FIG. 11, the possibility of outputting an image of an undesirable class is reduced by using an image class classification technique, and it is expected that the possibility of outputting an undesirable image can be reduced with relatively simple processing (relatively simple learning) compared to, for example, reducing the possibility of generating an undesirable image in an image generation process.

[0122] Sixth Embodiment 12 is a diagram illustrating an example of a procedure for a prompt learning method according to at least one embodiment. The prompt learning method illustrated in FIG. 12 includes extracting image features (step S621), extracting prompt features (step S622), calculating similarity (step S623), and updating a control prompt (step S624).

[0123] In extracting image features (step S621), the computer extracts an image feature vector, which is a feature vector of an input image that is an image of one of the classes in the image classification. In extracting the prompt features (step S622), the computer extracts a prompt feature vector, which is a feature vector of a combined prompt, which is data combining a base prompt, which is data indicating a class in image classification and an input image class, which is the class of the input image, and a control prompt, which is data to be updated. In calculating the similarity (step S623), the computer calculates the similarity between the prompt feature vector and the image feature vector.

[0124] In updating the control prompt (step S624), the computer updates the value of the control prompt so that the evaluation indicated by the evaluation value becomes better, using an evaluation function that outputs an evaluation value that indicates a worse evaluation the higher the similarity indicated by the similarity when the input image class is a suppression target class, which is a class for which image output should be suppressed, and that indicates a better evaluation the higher the similarity indicated by the similarity when the input image class is a class other than a suppression target class.

[0125] According to the prompt learning method shown in Figure 12, by inputting a prompt that has been learned using the prompt learning method shown in Figure 12 into a device that acquires an image in response to a prompt input, it is expected that the possibility of an undesirable image being output when an image is output in response to a prompt input can be reduced.

[0126] FIG. 13 is a diagram illustrating an example of a computer configuration according to at least one embodiment. In the configuration shown in FIG. 13, a computer 700 includes a CPU 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750.

[0127] One or more of the image acquisition device 100, the prompt learning device 300, the image acquisition device 400, the image acquisition device 610, and the prompt learning device 620, or a portion thereof, may be implemented in the computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main storage device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is performed by the interface 740, which has a communication function and performs communication under the control of the CPU 710. The interface 740 also has a port for the nonvolatile storage device 750, and reads and writes information from and to the nonvolatile storage device 750.

[0128] When the image acquisition device 100 is implemented in a computer 700, the operations of the processing unit 180 and each of its components are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0129] Furthermore, the CPU 710 allocates a storage area for the storage unit 170 in the main storage device 720 in accordance with the program. Communication with other devices by the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Display of images by the display unit 120 is performed by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving the user operations under the control of the CPU 710.

[0130] When the prompt learning device 300 is implemented in a computer 700, the operations of the processing unit 380 and each of its components are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0131] Furthermore, the CPU 710 allocates a storage area for the storage unit 170 in the main storage device 720 in accordance with the program. Communication with other devices by the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Display of images by the display unit 120 is performed by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving the user operations under the control of the CPU 710.

[0132] When the image acquisition device 400 is implemented in the computer 700, its operation is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0133] Furthermore, the CPU 710 allocates a storage area in the main memory device 720 for the image acquisition device 400 to perform processing in accordance with the program. Communication between the image acquisition device 400 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the image acquisition device 400 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0134] When the image acquisition device 610 is implemented in the computer 700, the operations of the image feature extraction unit 611, the prompt feature extraction unit 612, the similarity calculation unit 613, the control prompt update unit 614, and the output image acquisition unit 615 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0135] Furthermore, the CPU 710 allocates a storage area in the main memory device 720 for the image acquisition device 610 to perform processing in accordance with the program. Communication between the image acquisition device 610 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the image acquisition device 610 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0136] When the prompt learning device 620 is implemented in the computer 700, the operations of the image feature extraction unit 621, the prompt feature extraction unit 622, the similarity calculation unit 623, and the control prompt update unit 624 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0137] Furthermore, CPU 710 allocates a storage area in main memory 720 for processing by prompt learning device 620 in accordance with the program. Communication between prompt learning device 620 and other devices is carried out by interface 740, which has a communication function and operates under the control of CPU 710. Interaction between prompt learning device 620 and a user is carried out by interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of CPU 710 and accepting user operations via the input device.

[0138] One or more of the above-described programs may be recorded on nonvolatile recording medium 750. In this case, interface 740 may read the programs from nonvolatile recording medium 750. CPU 710 may then directly execute the programs read by interface 740, or may temporarily store the programs in main storage device 720 or auxiliary storage device 730 and then execute them.

[0139] Note that a program for executing all or part of the processing performed by image capture device 100, prompt learning device 300, image capture device 400, image capture device 610, and prompt learning device 620 may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to perform the processing of each part. Note that the term "computer system" here includes hardware such as an OS (Operating System) and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The program may be one that realizes part of the aforementioned functions, or may be one that can realize the aforementioned functions in combination with a program already stored in the computer system.

[0140] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs within the scope of the present invention. Furthermore, the above-described embodiments may be combined with other embodiments as appropriate.

[0141] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0142] (Appendix 1) an image feature extraction means for extracting an image feature vector, which is a feature vector of an input image that is an image of one of the classes in image classification; a prompt feature extraction means for extracting a prompt feature vector, which is a feature vector of a combined prompt, which is data obtained by combining a base prompt, which is data indicating a class in the image classification and an input image class, which is a class of the input image, and a control prompt, which is data to be updated; a similarity calculation means for calculating a similarity between the prompt feature vector and the image feature vector; a control prompt updating means for updating the value of the control prompt so as to improve the evaluation indicated by the evaluation value, using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the degree of similarity increases when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the degree of similarity increases when the input image class is a class other than the suppression target class; an output image acquisition means for acquiring an image using a prompt feature vector according to the updated control prompt; An image acquisition device comprising:

[0143] (Appendix 2) the output image acquisition means generates an image using a prompt feature vector based on the updated control prompt. 10. The image acquisition device of claim 1.

[0144] (Appendix 3) the output image acquisition means acquires an image by image search using an image generated using a prompt feature vector based on the updated control prompt. 3. The image acquisition device of claim 2.

[0145] (Appendix 4) the output image acquisition means acquires an image by image search using the prompt feature vector based on the updated control prompt; 4. An image acquisition device according to any one of claims 1 to 3.

[0146] (Appendix 5) The input image is an image obtained by an image search using a keyword that indicates the input image class. 5. An image acquisition device according to any one of claims 1 to 4.

[0147] (Appendix 6) an image feature extraction means for extracting an image feature vector, which is a feature vector of an input image that is an image of one of the classes in image classification; a prompt feature extraction means for extracting a prompt feature vector, which is a feature vector of a combined prompt, which is data obtained by combining a base prompt, which is data indicating a class in the image classification and an input image class, which is a class of the input image, and a control prompt, which is data to be updated; a similarity calculation means for calculating a similarity between the prompt feature vector and the image feature vector; a control prompt updating means for updating the value of the control prompt so as to improve the evaluation indicated by the evaluation value, using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the degree of similarity increases when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the degree of similarity increases when the input image class is a class other than the suppression target class; A prompt learning device comprising:

[0148] (Appendix 7) The input image is an image obtained by an image search using a keyword that indicates the input image class. 7. The prompt learning device of claim 6.

[0149] (Appendix 8) The computer Extracting an image feature vector that is a feature vector of an input image that is an image of one of the classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt that is data obtained by combining a base prompt that is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt that is data to be updated; Calculating a similarity between the prompt feature vector and the image feature vector; updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the similarity increases when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the similarity increases when the input image class is a class other than the suppression target class; Acquire an image using the prompt feature vector according to the updated control prompt. An image acquisition method comprising:

[0150] (Appendix 9) obtaining the image includes generating, by the computer, an image using a prompt feature vector according to the updated control prompt; The image acquisition method described in Appendix 8.

[0151] (Appendix 10) and acquiring the image by an image search using an image generated by the computer using a prompt feature vector based on the updated control prompt. 10. The image acquisition method described in Appendix 9.

[0152] (Appendix 11) and acquiring the image by the computer using an image search with a prompt feature vector based on the updated control prompt. 11. An image acquisition method according to any one of appendices 8 to 10.

[0153] (Appendix 12) The input image is an image obtained by an image search using a keyword that indicates the input image class. 12. An image acquisition method according to any one of appendices 8 to 11.

[0154] (Appendix 13) The computer Extracting an image feature vector that is a feature vector of an input image that is an image of one of the classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt that is data obtained by combining a base prompt that is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt that is data to be updated; Calculating a similarity between the prompt feature vector and the image feature vector; When the input image class is a suppression target class, which is a class for which image output should be suppressed, the higher the similarity indicated by the similarity, the worse the evaluation, and when the input image class is a class other than the suppression target class, the higher the similarity indicated by the similarity, the better the evaluation is, using an evaluation function that outputs an evaluation value that indicates a better evaluation, updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better. Prompt learning methods that include:

[0155] (Appendix 14) The input image is an image obtained by an image search using a keyword that indicates the input image class. The prompt learning method described in Appendix 13.

[0156] (Appendix 15) On the computer, Extracting an image feature vector that is a feature vector of an input image that is an image of one of the classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt that is data obtained by combining a base prompt that is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt that is data to be updated; calculating a similarity between the prompt feature vector and the image feature vector; updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the similarity becomes higher when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the similarity becomes higher when the input image class is a class other than the suppression target class; acquiring an image using a prompt feature vector according to the updated control prompt; A program that executes the following.

[0157] (Appendix 16) In the step of acquiring the image, the program causes the computer to generate an image using a prompt feature vector according to the updated control prompt. 16. The image acquisition method according to claim 15.

[0158] (Appendix 17) In the step of acquiring the image, the program causes the computer to acquire the image by image search using an image generated using a prompt feature vector based on the updated control prompt; 17. The program according to claim 16,

[0159] (Appendix 18) In the step of acquiring the image, the program causes the computer to acquire the image by image search using the prompt feature vector based on the updated control prompt; 18. The program according to any one of appendices 15 to 17,

[0160] (Appendix 19) The input image is an image obtained by an image search using a keyword that indicates the input image class. 19. The program of any one of appendices 15 to 18.

[0161] (Appendix 20) On the computer, Extracting an image feature vector that is a feature vector of an input image that is an image of one of the classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt that is data obtained by combining a base prompt that is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt that is data to be updated; calculating a similarity between the prompt feature vector and the image feature vector; updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the similarity becomes higher when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the similarity becomes higher when the input image class is a class other than the suppression target class; A program that executes the following.

[0162] (Appendix 21) The input image is an image obtained by an image search using a keyword that indicates the input image class. 20. The program described in Appendix 20. [Explanation of symbols]

[0163] 10. Image Acquisition System 100, 400, 610 Image acquisition device 110 Communications Department 120 Display section 130 Operation input section 170 Storage section 180, 380 processing section 181 Input image acquisition unit 182 Base prompt acquisition unit 183 Control prompt setting section 184, 611, 621 Image feature extraction section 185, 612, 622 Prompt feature extraction section 186, 613, 623 Similarity calculation unit 187 Class Output Unit 188 Loss Calculation Department 189, 614, 624 Control prompt update section 190, 615 Output image acquisition unit 191 Image output unit 300, 620 Prompt learning device 392 Prompt Output Section

Claims

1. an image feature extraction means for extracting an image feature vector, which is a feature vector of an input image that is an image of one of the classes in image classification; a prompt feature extraction means for extracting a prompt feature vector, which is a feature vector of a combined prompt, which is data obtained by combining a base prompt, which is data indicating a class in the image classification and an input image class, which is a class of the input image, and a control prompt, which is data to be updated; a similarity calculation means for calculating a similarity between the prompt feature vector and the image feature vector; a control prompt updating means for updating the value of the control prompt so as to improve the evaluation indicated by the evaluation value, using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the degree of similarity increases when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the degree of similarity increases when the input image class is a class other than the suppression target class; an output image acquisition means for acquiring an image using a prompt feature vector according to the updated control prompt; An image acquisition device comprising:

2. the output image acquisition means generates an image using a prompt feature vector based on the updated control prompt. The image acquisition device of claim 1 .

3. the output image acquisition means acquires an image by image search using an image generated using a prompt feature vector based on the updated control prompt. The image acquisition device of claim 2 .

4. the output image acquisition means acquires an image by image search using the prompt feature vector based on the updated control prompt; The image acquisition device of claim 1 .

5. The input image is an image obtained by an image search using a keyword that indicates the input image class. The image acquisition device of claim 1 .

6. an image feature extraction means for extracting an image feature vector, which is a feature vector of an input image that is an image of one of the classes in image classification; a prompt feature extraction means for extracting a prompt feature vector, which is a feature vector of a combined prompt, which is data obtained by combining a base prompt, which is data indicating a class in the image classification and an input image class, which is a class of the input image, and a control prompt, which is data to be updated; a similarity calculation means for calculating a similarity between the prompt feature vector and the image feature vector; a control prompt updating means for updating the value of the control prompt so as to improve the evaluation indicated by the evaluation value, using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the degree of similarity increases when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the degree of similarity increases when the input image class is a class other than the suppression target class; A prompt learning device comprising:

7. The computer Extracting an image feature vector that is a feature vector of an input image that is an image of one of the classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt that is data obtained by combining a base prompt that is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt that is data to be updated; Calculating a similarity between the prompt feature vector and the image feature vector; updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the similarity increases when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the similarity increases when the input image class is a class other than the suppression target class; Acquire an image using the prompt feature vector according to the updated control prompt. An image acquisition method comprising:

8. The computer Extracting an image feature vector that is a feature vector of an input image that is an image of one of the classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt that is data obtained by combining a base prompt that is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt that is data to be updated; Calculating a similarity between the prompt feature vector and the image feature vector; When the input image class is a suppression target class, which is a class for which image output should be suppressed, the higher the similarity indicated by the similarity, the worse the evaluation, and when the input image class is a class other than the suppression target class, the higher the similarity indicated by the similarity, the better the evaluation is, using an evaluation function that outputs an evaluation value that indicates a better evaluation, updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better. Prompt learning methods that include:

9. On the computer, Extracting an image feature vector that is a feature vector of an input image that is an image of one of the classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt that is data obtained by combining a base prompt that is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt that is data to be updated; calculating a similarity between the prompt feature vector and the image feature vector; updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the similarity becomes higher when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the similarity becomes higher when the input image class is a class other than the suppression target class; acquiring an image using a prompt feature vector according to the updated control prompt; A program that executes the following.

10. On the computer, Extracting an image feature vector that is a feature vector of an input image that is an image of one of the classes in image classification; extracting a prompt feature vector that is a feature vector of a combined prompt that is data obtained by combining a base prompt that is data indicating the class in the image classification and the input image class that is the class of the input image, and a control prompt that is data to be updated; calculating a similarity between the prompt feature vector and the image feature vector; updating the value of the control prompt so that the evaluation indicated by the evaluation value becomes better using an evaluation function that outputs an evaluation value that indicates a worse evaluation as the similarity indicated by the similarity becomes higher when the input image class is a suppression target class that is a class for which image output should be suppressed, and that indicates a better evaluation as the similarity indicated by the similarity becomes higher when the input image class is a class other than the suppression target class; A program that executes the following.

Images