Information processing device, information processing method, and program

Abstract

Even when a user uses a different model than the one they previously used, it makes it easier to obtain similar generation results to those obtained with the previously used model. [Solution] The first input information is input into a first machine learning model to obtain a first generation result. The first input information is input into a second machine learning model different from the first machine learning model to obtain a second generation result. A first change in an item that varies depending on the difference in the machine learning models used for generation between the first generation result and the second generation result is detected. When the first change is detected, first auxiliary information is generated based on the first change, which includes information for changing the input information to be input into the second machine learning model from the first input information.

Description

[Technical Field]

[0001] This invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] One definition of generative AI, a major trend in recent artificial intelligence (AI) technology, is a technique that uses a pre-trained Deep Neural Network (DNN) to generate new data not present in the pre-trained data in response to input such as text prompts. Generative AI is rapidly developing, supported by advancements in deep generative models and multimodal foundational models. Recent deep generative models, such as diffusion models, can generate very high-quality data. Furthermore, multimodal foundational models such as Contrastive Language-Image Pre-training (CLIP, Non-Patent Literature 1) can perform highly sophisticated mapping of different modalities, such as language and images, thanks to their high Zero (Few)-Shot performance.

[0003] Generative AI tasks include various types that generate output Y from input X, and such tasks are called X-to-Y tasks. By using X-to-Y tasks, various modal conversions can be performed with high performance, such as generating content like images, videos, or 3D data using language, or image style transformation. Tasks where the input is a text prompt include tasks such as Text-to-Image, Text-to-Video, and Text-to-3D, which generate arbitrary images, videos, or 3D data specified by the user. There are also other tasks such as Image-to-Image tasks that perform image style transformation or inpainting.

[0004] A major difference between these generative AI tasks and conventional AI is their high level of practicality, as they can interactively generate high-quality, creative content using natural language that everyone uses. Prompt engineering makes it easier to obtain the desired image without having to retrain the model.

[0005] Due to their high practicality, generative AI services have rapidly increased, and new services with features incorporating rapid technological advancements are constantly emerging. In this rapidly changing market, frequently updating models has become one of the important measures for creating better services. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Learning transferable visual models from natural language supervision, Radford et al., ICML2021 [Overview of the project] [Problems that the invention aims to solve]

[0007] Users of generative AI spend a considerable amount of time adjusting inputs such as text prompts to obtain the desired image. However, when the model used is changed, such as before or after an X-to-Y model upgrade, the style of the generated result obtained from the same input may change. Even if the old model is unusable due to the upgrade, readjusting the prompts is time-consuming, even if the goal is to generate a new generated result Y with a similar style to the generated result Y previously produced using the old model.

[0008] The present invention aims to make it easier for users to obtain generation results similar to those obtained with previously used models, even when using a different model than the one previously used. [Means for solving the problem]

[0009] To achieve the object of the present invention, for example, an information processing device according to one embodiment has the following configuration: a first acquisition means for inputting first input information into a first machine learning model to obtain a first generation result; a second acquisition means for inputting the first input information into a second machine learning model different from the first machine learning model to obtain a second generation result; a detection means for detecting a first variation in an item that varies according to the difference between the first generation result and the second generation result in the machine learning model used for generation; and a generation means for generating auxiliary information, including information for changing the input information to be input to the second machine learning model from the first input information, based on the first variation when the first variation is detected. [Effects of the Invention]

[0010] Even when a user uses a different model than the one they previously used, it makes it easier to obtain similar generation results to those obtained with the previously used model. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram showing an example of the hardware configuration of a system including an information processing device. [Figure 2] A block diagram showing an example of the functional configuration of the information processing device according to Embodiment 1. [Figure 3] A flowchart showing an example of information processing according to Embodiment 1. [Figure 4] A diagram illustrating an example of detecting object movement. [Figure 5] A block diagram showing an example of the functional configuration of the information processing device according to Embodiment 2. [Figure 6] A flowchart showing an example of information processing according to Embodiment 2. [Figure 7] A flowchart showing an example of information processing according to Embodiment 3. [Figure 8] A diagram illustrating the list generated in response to detected objects. [Figure 9]A diagram for explaining an example of detecting fluctuations in the color of an object. [Figure 10] A diagram for explaining an example of a database of generation history information.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate explanations are omitted.

[0013] <Embodiment 1> The information processing apparatus according to the present embodiment inputs first input information including text information into a first machine learning model and a second machine learning model, respectively, and obtains a first generation result and a second generation result as respective outputs. Next, the information processing apparatus detects fluctuations in items that vary according to the machine learning model used for generation between the first generation result and the second generation result. When such fluctuations are detected, the information processing apparatus generates auxiliary information used to change the input information to the second machine learning model from the first input information based on the detected fluctuations. Hereinafter, such an information processing apparatus will be described.

[0014] In the X-to-Y task, when the machine learning model used is changed due to version upgrade or the like, the generation result Y for the same input X may vary. Here, in such a case, a change in X is recommended to assist the user in obtaining a similar generation result. In the following, the machine learning model that outputs an output for an input may sometimes be simply referred to as a "model", and when simply denoted as input X and output (generation result) Y, it refers to the input and output of such a model.

[0015] In this embodiment, the input X input to the model includes text information such as text prompts. The input X may also include other information such as style images, configuration images, or bounding box (BB) information that specifies the layout. The type of data (modality) of the generated result Y output by the model is not particularly limited and may be, for example, an image, a video, or 3D. The model according to this embodiment will be described below as performing a Text-to-Image task that takes text information as input and outputs image data.

[0016] Even with the same input X, the resulting Y may vary depending on the model used. Examples of variable items include the color tone of the image, the objects that make up the image, the composition (position, size, or relative positions of objects), and changes in the degree of blur. In this embodiment, we will describe an example in which changes in the objects that make up the image are detected as variable items.

[0017] <Hardware Configuration> Embodiment 1 describes an example in which each process is executed using a system consisting of a user terminal connected to a network and a server that provides services. The information processing device according to this embodiment is implemented as the server 103 that provides the services.

[0018] Figure 1(a) shows an example of the configuration of a system including a server 103 according to this embodiment. The server 103 is connected to a client (user terminal) 101 via a network 102.

[0019] Figure 1(b) is a block diagram showing an example of the hardware configuration of client 101 according to this embodiment. Client 101 has a CPU 101-1, ROM 101-2, RAM 101-3, storage unit 101-4, communication unit 101-5, input unit 101-6, and display unit 101-7, and each component is connected by a system bus.

[0020] CPU 101-1 controls the entire device by executing a control program stored in ROM 101-2. RAM 101-3 temporarily stores various data from each component. RAM 101-3 also expands the program to make it executable by CPU 101-1. Storage unit 101-4 stores data to be processed by server 103 according to this embodiment, and stores generation history information or temporarily stores data generated during information processing. Here, generation history information includes input information such as text prompts used by the user to generate data, model information indicating which model was used, and information including the generation result. As the medium for storage unit 101-4, HDD, flash memory, or various optical media can be used. Communication unit 101-5 performs communication between client 101 and server 103 via network 102. Input unit 101-6 accepts input such as text prompts from the user. Display unit 101-7 displays input to the model and generation results, past generation history information, or auxiliary information for prompt changes by the system.

[0021] Figure 1(c) is a block diagram showing an example of the hardware configuration of server 103. Server 103 has a CPU 103-1, ROM 103-2, RAM 103-3, storage unit 103-4, and communication unit 103-5, and each component is connected by a system bus. The CPU 103-1, ROM 103-2, RAM 103-3, storage unit 103-4, and communication unit 103-5 can be configured in the same way as the CPU 101-1, ROM 101-2, RAM 101-3, storage unit 101-4, and communication unit 101-5, so a description of their functions is omitted here.

[0022] Furthermore, client 101 may be equipped with hardware specialized for AI processing, such as a Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), or a chip dedicated to AI processing, in order to perform Deep Neural Network (DNN) processing at high speed. The same applies to server 103.

[0023] <Processing details> Figure 2 shows an example of the functional configuration of the server 103 according to this embodiment. The server 103 includes a storage unit 200, a model management unit 201, an acquisition unit 202, a first generation result acquisition unit (first result acquisition unit) 203, an output unit 204, a control unit 205, a second generation result acquisition unit (second result acquisition unit) 206, a detection unit 207, an auxiliary information generation unit 208, and a reception unit 209.

[0024] In this embodiment, the same input information, namely first input information, is input to a first machine learning model and a second machine learning model that is different from the first machine learning model. Here, the second machine learning model is an upgraded (updated) version of the first machine learning model. Figures 2(a) and 2(b) show the data flow between the functional parts of the pre-update model and the post-update model, respectively. Below, we will describe examples of the first image generation process (a) executed when the first input information is input to the pre-update model and the second image generation process (b) executed when the first input information is input to the post-update model.

[0025] [First image generation process (a)] This section describes the processing performed by each functional unit of the server 103 in the first image generation process (a). The storage unit 200 stores generation history information and models. In this embodiment, the storage unit 200 stores multiple models, one of which is used as the model to be processed in subsequent processing. The model management unit 201 manages information indicating the version of the model stored in the storage unit 200, and information indicating whether the user can use the model. Hereinafter, this information managed by the model management unit 201 may be referred to as "model management information".

[0026] The acquisition unit 202 acquires input information to be input to the first model as first input information. In this embodiment, the first input information is assumed to be a text prompt. The acquisition unit 202 may acquire the first input information based on information entered by the user in the input unit 101-6 (or, for example, via the reception unit 209 described later). Here, the acquisition unit 202 acquires model management information from the model management unit and, based on the model management information, can acquire a model from among the models stored in the storage unit 200 that is permitted for use by the user as the first model. Here, it is assumed that the user executing the process is identified through the login process or by identifying the connected client.

[0027] The first result acquisition unit 203 inputs the first input information acquired by the acquisition unit 202 into the first model and acquires the output for that input as the first generation result.

[0028] The output unit 204 outputs the first generation result acquired by the first result acquisition unit 203 as generation history information, along with input information such as text prompts used for data generation, and model information indicating which model was used for data generation, and the storage unit 200 stores this output. The output unit 204 may also output the generation result or generation history information to a display device such as the display unit 101-7.

[0029] [Second image generation process (b)] Next, the processing performed by each functional unit of the server 103 in the second image generation process (b) will be explained. The control unit 205 controls the operation of the acquisition unit 202 and the detection unit 207 based on the model management information. Specifically, when the model is upgraded from the first model to the second model, the control unit 205 can control each functional unit to acquire the first input information and the second model and generate the second generation result.

[0030] The acquisition unit 202 acquires the second model. Here, the acquisition unit 202 acquires model management information and, based on the model management information, can acquire the second model, which is the first model that has been updated.

[0031] The second result acquisition unit 206 inputs the first input information acquired by the acquisition unit 202 into the second model and acquires the output for that input as the second generation result. The acquisition unit 202 can also acquire second input information (described later) obtained by modifying the first input information, and the second result acquisition unit 206 can input this second input information into the second model and acquire a third generation result.

[0032] The detection unit 207 detects the variation in items (variable items) that change according to the difference in the model used for generation between the first generation result and the second generation result. Here, items that change according to the model version upgrade are used as variable items, and it is assumed that such variable items are pre-configured. A detailed explanation of variable items will be given later, but for example, if "number of objects" is set as a variable item, the variation in the number of predetermined objects to be detected included in each output is detected. Note that the number of variable items is not particularly limited, and there may be only one or multiple variable items.

[0033] The auxiliary information generation unit 208 generates auxiliary information when a variation is detected, based on the variation detection result detected by the detection unit 207. The auxiliary information according to this embodiment is generated based on the variation detected by the detection unit 207 and includes information for changing the input information to be input to the second model from the first input information. Here, the auxiliary information generation unit 208 can generate auxiliary information such that the variation between the first generation result and the third generation result obtained by inputting the second input information, which is the first input information modified based on the auxiliary information, into the second model is smaller than the variation between the first generation result and the second generation result. In particular, here, the auxiliary information used is information indicating a policy for changing the input, which assists in how to change the first input information in order to obtain a generation result similar to the first generation result when input to the second model. For example, text information that encourages suppressing the variation between the first generation result and the second generation result can be used as auxiliary information, but a detailed explanation will be given later with reference to Figure 4, etc. The auxiliary information is presented to the user by being displayed on the display unit 101-7.

[0034] The reception unit 209 receives user input after generating and presenting auxiliary information to the user. Then, the reception unit generates second input information based on the first input information and the user input received after the presentation to the user. The second input information is stored in the storage unit 200.

[0035] Hereinafter, an example of information processing performed by the server 103 according to this embodiment will be described with reference to Figure 3. The processing shown in Figure 3 is just one example, and the server 103 does not need to perform all the processing described using this flowchart. The flowchart in Figure 3(a) shows the processing by the first model (first image generation processing (a)), and the flowchart in Figure 3(b) shows the processing by the second model (second image generation processing (a)).

[0036] [Processing by the first model] Figure 3(a) is a flowchart showing an example of the process when the first image generation process (a) is executed using the first model. The process shown in Figure 3(a) is executed, for example, when the system receives a request from client 101 to execute an image generation process based on user input.

[0037] In S3001, the acquisition unit 202 acquires model management information. Here, the model management information is assumed to include a list of models currently available to the user and version information for those models.

[0038] In S3002, the acquisition unit 202 acquires an available model as the first model based on the model management information acquired in S3001.

[0039] In S3003, the acquisition unit 202 acquires the first input information. Here, the acquisition unit 202 may acquire the text prompt entered by the user into the input device as the first input information via the reception unit 209.

[0040] In S3004, the first result acquisition unit 203 inputs the first input information acquired in S3003 into the first model to obtain the first generation result. The first generation result may be generated as multiple results using different random seed values, or as just one result.

[0041] In S3005, the output unit 204 outputs the first generation result obtained in S3004 as generation history information, along with input information such as text prompts used for data generation, and model information indicating which model was used for data generation, and then terminates processing. Here, the storage unit 200 stores the generation history information. The generation history information may be displayed on a display device (for example, a display unit 101-7 provided by the client 101), and may be displayed together with other information such as input information or model information.

[0042] [Processing by the second model] Figure 3(b) is a flowchart showing an example of the process when the second image generation process (b) is performed using the second model and auxiliary information is generated. The process shown in Figure 3(b) is executed, for example, when the execution process for image generation is received from client 101 based on user input. The process shown in Figure 3(b) may also be executed periodically, or when a start command is given by the user.

[0043] In S3101, the acquisition unit 202 acquires model management information for each model. Here, it is assumed that the model management information includes information on the date and time of version upgrades for each version of the model.

[0044] In S3102, the acquisition unit 202 checks whether there is a model that has been updated. If the model has been updated, the process proceeds to S3103; otherwise, the process returns to S3101.

[0045] The process of checking whether a model has been updated can be performed using any method for detecting model updates. For example, in the loop processing from S3101 to S3102, the acquisition unit 202 can check whether a model has been updated by comparing the model management information acquired in S3101 of the previous loop with the model management information acquired in S3101 of the current loop. Alternatively, the acquisition unit 202 may record information indicating the latest model version included in the model management information when it confirms that a model has been updated, and if the version confirmed in S3102 is not a recorded version, it may be assumed that the model has been updated. Furthermore, the loop from S3101 to S3102 may be performed each time the presence or absence of updates is checked for all managed models, or the loop may be executed separately for each model.

[0046] In S3103, the acquisition unit 202 acquires the model that has been updated as the model to be processed and as a second model, and further acquires the generation history information of the said model.

[0047] In S3104, the acquisition unit 202 acquires the first input information that was entered into the model before the update of the model to be processed, based on the generation history information acquired in S3103.

[0048] In S3105, the second result acquisition unit 206 inputs the first input information acquired in S3104 into the second model and acquires a second generation result. The second generation result may be generated as multiple results using different random seed values, or it may be generated as a single result.

[0049] In S3106, the detection unit 207 obtains a first generation result, which is the result of inputting the first input information into the first model. Here, the detection unit 207 obtains the first generation result included in the generation history information obtained in S3103, but the data to be used as the first generation result may be input by user operation.

[0050] In S3107, the detection unit 207 detects the variation in the variation items between the second generation result obtained in S3105 and the first generation result obtained in S3106, and outputs it as a variation detection result.

[0051] The following explanation will describe an example of detecting changes in the objects constituting an image as a variable item, with reference to Figure 4. Figures 4(a) and 4(b) show the first and second generation results, respectively. Here, both the first and second generation results are images generated by inputting the text prompt "A frog is standing in a grassland under a blue sky" as the first input information to the model. In Figure 4(b), the second result acquisition unit 206 generates three different images using different random seed values ​​(for example, in S3105), which are shown as the second generation results. Thus, there may be multiple second generation results, or there may be only one.

[0052] The detection unit 207 first uses an object detector, such as a DNN, to detect objects in an image, and uses this detector to detect objects in the image from the first generation result and the second generation result, respectively. Next, the detection unit 207 outputs the detected objects and their number as a list. An example of the list output here is shown in Figure 8(a). Figure 8(a-1) shows the list for the first generation result (image 1-1) shown in Figure 4(a), where each row of the list represents each object, and each column represents the image name, object category, location, and size in which the object was detected. Figure 8(a-2) shows the list for the second generation result (image 2-1, image 2-2, and image 2-3) shown in Figure 4(a-2), where each row of the list represents each object, and each column represents the image name, object category, location, and size in which the object was detected.

[0053] Next, the detection unit 207 aggregates the number of objects in each image of the first and second generation results, categorized by type, based on the list of object detection results. If multiple images are generated for each generation result, the total number of objects detected in each category in those images is averaged using the total number of generated images, and this average is aggregated in the list as the "number of detected objects".

[0054] Figure 8(b) shows an example of the number of objects detected for each category. Figure 8(b-1) shows the number of objects detected corresponding to the list shown in Figure 8(a-1), and Figure 8(b-2) shows the number of objects detected corresponding to the list shown in Figure 8(b-1). In this example, one "frog" and one "mountain" are detected from the first generated image, and an average of one "frog," an average of 2 / 3 "mountains," and an average of one "pond" are detected from the second generated three images.

[0055] Here, we will explain the detection of variation using the number of objects detected within an image as the variation item "number of objects." However, the process is not limited to this if it is possible to perform an evaluation related to the number of objects in a similar manner. For example, if multiple images are generated as a result, the number of images in which objects of the corresponding category are detected can be calculated, and the detection frequency in which such detections occur can be used as the evaluation value for the number of objects. Even when performing such processing, in the example shown in Figure 4(b), since mountains are detected in 2 out of 3 images, the evaluation value shown in Figure 8(b-2) will be 2 / 3.

[0056] Next, the detection unit 207 (in S3108) determines whether or not a change has occurred based on these aggregated results. Here, the detection unit 207 calculates the increase or decrease in the number of objects per image for each object category (category c), and determines whether or not a change has occurred based on such increase or decrease figures. Here, for category c, the number of objects detected in the first generation result and the second generation result are N1 c , N2 c It is represented as follows. The detection unit 207 controls the increase or decrease in detection frequency Δ=N2 c -N1 c The system calculates Δ and determines that there is a change if there are M or more object categories where Δ is greater than or equal to the threshold. Here, the threshold only needs to be greater than 0, and M only needs to be greater than or equal to 1, but these values ​​can be arbitrarily set according to the desired conditions. Figure 8(b-3) shows the value of Δ for such a change (in this example, the threshold for the increase / decrease is set to 1). In Figure 8(b-3), Δ is 0 for the categories "frog" and "mountain", but Δ is 1 for the category "pond", so it is determined that a change has occurred.

[0057] In S3108, the detection unit 207 determines whether or not a fluctuation has occurred based on the fluctuation detection result in S3107, as described above. If a fluctuation has occurred, the process proceeds to S3109; otherwise, the process shown in Figure 3(a) ends.

[0058] In S3109, the auxiliary information generation unit 208 generates auxiliary information. As described above, auxiliary information includes information for modifying the input information to be input to the second model from the first input information. Here, the auxiliary information used is information indicating the input modification policy, which helps in how to modify the first input information in order to obtain a generation result similar to the first generation result when input to the second model.

[0059] In the example shown in Figure 4, based on the object detection results shown in Figure 8, it was determined that the number of "pond" outputs (generation frequency) in the second generation result had changed compared to the first generation result. Therefore, the output unit 204 generates auxiliary information suggesting, for example, that a positive or negative prompt be input for objects in the category where the changed items have changed, as an input change policy (labeled "prompt change policy" in Figure 4). In this embodiment, a (positive) prompt is a prompt included in the input to the model that specifies the content to be included in the generated image when generating an image. In this embodiment, a negative prompt is a prompt included in the input to the model that specifies the content to be excluded from the generated image when generating an image in a generation AI using prompts such as Text-to-Image. Here, negative prompts are assumed to be input separately from positive prompts. For example, the output unit 204 can generate auxiliary information suggesting that a negative prompt be input for objects in the category where the generation frequency has increased, as these objects may be unnecessary for the user.

[0060] In the example in Figure 4, since the frequency of "pond" generation has increased, a message such as "Model v2 (updated model) tends to generate 'pond' more often than Model v1 (pre-update model). If you do not need it, please enter 'pond' in the negative prompt input field" is output. For objects whose output count has decreased, the user may be presented with a policy to change the input, such as adding a sentence to the positive prompt that increases the number of that object, such as "A landscape with 'XX'" (XX is the name of the object category). The sentences output here may be predefined or sequentially generated, and are not limited to these types of wording. Furthermore, in prompt input, as shown in Figure 4(c), separate inputs may be possible on the user interface (UI) for each type of prompt (for example, positive prompts and negative prompts).

[0061] Furthermore, if there are specific variable items that are known in advance to be difficult to suppress by changing only the text prompt, auxiliary information may be generated to prompt additional changes to inputs other than the text prompt, in response to the detection of a change in those variable items. Regarding the positional relationships of objects (for example, when outputting "generate XX to the right of YY"), it may be recommended that the user specify the generation position of such objects in the image using a Big Block (BB). In this case, the position of the BB may be set based on user input, or it may be set by presenting object BBs detected from the image as candidates and allowing the user to select from among them.

[0062] Furthermore, if there is a change in the position or orientation of an object, auxiliary information may be generated prompting the user to specify a reference image (composition image) that represents the composition of the image. Also, if there is a change in style, auxiliary information may be generated prompting the user to specify a reference image (style image) to be used for style conversion. In such cases, the output unit 204 can generate auxiliary information prompting the user to input the reference image into the (Text+Image)-to-Image model or the Image-to-Image model. In addition, the output unit 204 may generate auxiliary information that prompts fine-tuning of the model, such as LoRA training, using the first generated image generated by the first model. Alternatively, an Image-to-Image model that takes the first and second generation results as inputs and outputs a reference image (style image or composition image) that bridges the gap between them may be pre-trained and used as input to the (Text+Image)-to-Image model.

[0063] In S3110, the output unit 204 displays auxiliary information on the display device. This auxiliary information may be displayed together with input information or model information. Figure 4(c) shows an example of how auxiliary information is presented. In the example in Figure 4(c), the first generation result and the second generation result are displayed side by side, and the auxiliary information described above, "In model v2, 'pond' tends to be generated more often than in the generated image of model v1. If it is not needed, please enter 'pond' in the negative prompt input field," is also displayed. If there are multiple categories of objects for which the number of outputs has changed, the auxiliary information may be presented as options for which of these objects to increase / decrease the number of outputs. In such cases, the user can click on the option to input a positive or negative prompt for the selected object. In Figure 4(c), the prompt change policy (auxiliary information) and the prompt input field are displayed on the same screen, but the display layout is not limited to this. As shown in Figure 4(c), displaying the first and second generation results together with auxiliary information makes it possible to present the intent of the prompt change policy to the user in a more easily understandable way.

[0064] In S3111, the reception unit 209 receives a change in input by the user and obtains the information that has been changed from the first input information as the second input information. Here, the reception unit 209 receives the change in input information from the user after inputting auxiliary information for display in S3110. For example, the reception unit 209 can display a UI that presents auxiliary information to the user and accepts the changes to the first input information, and generate the second input information by changing the first input information based on the information entered on the UI. Such a UI is shown in Figure 4(c), for example. The change in input by the user here may be confirmed by pressing the input change button while "pond" is entered in the negative prompt field in Figure 4(c), but the UI is not limited to this.

[0065] In S3112, the second result acquisition unit 206 inputs the second input information acquired in S3111 into the second model and acquires the third generation result.

[0066] In S3113, the output unit 204 links the third generation result to the generation history information corresponding to the first generation result, and the storage unit 200 stores the said generation history information. Here, the input information and model information used to generate the third generation result are also stored as generation history information. In addition, the first generation result, the second generation result, and the third generation result may be presented side by side upon request from the user.

[0067] In this explanation, the processes from S3103 onward are executed assuming that the models for which updates have been confirmed from among the managed models are the targets of processing. However, this is not a limitation, as long as processing can be performed using the models before and after the update. For example, the models to be processed may be specified in advance (for example, as the models that have completed the processing in Figure 3(a)), and S3101 to S3102 may be used to check whether or not the models in question have been updated. Furthermore, if there are multiple models to be processed (for example, if updates to multiple models are confirmed in S3102), the subsequent processing will be executed separately for each of them.

[0068] <Effects of Embodiment 1> With this configuration, it is possible to detect changes in the variable items between the first and second generation results and generate auxiliary information for changing the input information to be input to the second model based on those changes. In particular, when the generation result for the same input changes due to a model version upgrade, it is possible to present auxiliary information that suggests a policy for changing the input information so that the same generation result as before the version upgrade can be obtained, based on those changes. Therefore, even if the user uses a different model than the one previously used, it becomes easier to obtain the same generation result as the one obtained with the previously used model.

[0069] <Variation Example 1-1> In Embodiment 1, the case where the number of objects is used as a variable item corresponding to a model version upgrade was described. In this modification, the case where the color of the objects is used as a variable item will be described. In the following, the detection unit 207 will be described as calculating the average value of the hue from the pixel values ​​(RGB) of the object region and detecting the variation for each object. However, the evaluation of the detected color variation is not limited to this method, and for example, the variation may be detected using the average value of the hue of the entire image. In addition, as for color, coordinates in the color space may be used instead of hue. Here, only the points that differ from the description above in Embodiment 1 will be explained.

[0070] In S3107 according to this modification example, the detection unit 207 detects the variation between the second generation result obtained in S3105 and the first generation result obtained in S3106, and outputs it as a variation detection result.

[0071] Hereinafter, an example of detecting the variation in the color of an object as a variation item will be described with reference to FIG. 9. FIGS. 9(a-1) and 9(a-2) show Image 1 which is the first generation result and Image 2 which is the second generation result. The detection unit 207 first uses an object detector to detect the object in the image from each of the first generation result and the second generation result, and outputs a list of object detection results as shown in FIG. 9(b). In this example, the object area is detected by the object detector and output as the position and size of the BB area surrounding the object, but the detection of the object area is not particularly limited in this way, and it may be performed by a known process for detecting an object in an image. For example, the object area may be specified using a region divider and output as a segmentation mask. FIG. 9(c) shows an enlarged view of the BB area detected from the first generation result.

[0072] Next, the detection unit 207 calculates the hue from the RGB pixel values of each pixel in the object area for each of the first generation result and the second generation result, and calculates the average value Hue t of the hue of the entire object area. The Hue t calculated in this way is shown in FIG. 9(d-1) for Image 1 and in (d-2) for Image 2. Hue t can be calculated, for example, according to the following formula (1).

Equation

[0073] Here, N t is the number of pixels in the object area, and Hue i t represents the hue calculated from the RGB value of the i-th pixel of image t. N tFor example, the number of pixels within the bounding box (W t ×H t The calculation method is not particularly limited and may be calculated as follows: it may be calculated by sequentially counting the pixels with i=0 as the top-left pixel of the bounding box. If the object region is identified by segmentation, the hue may be calculated for the pixels in the region of the segmentation mask. Furthermore, as in Embodiment 1, if multiple objects of the same category are detected, the average value of the hue calculated for each object may be taken to calculate one average hue for each category in each image.

[0074] In S3108, the detection unit 207 determines whether or not a change has occurred based on these aggregated results. Here, the detection unit 207 calculates the difference ΔHue between Hue1 and Hue2, which are the average values ​​of hues of objects in the same category detected in the first and second generation results, for each object category. If there are M or more object categories where the difference ΔHue is greater than or equal to the threshold, it is determined that a change has occurred. Here, the threshold only needs to be greater than 0, and M only needs to be 1 or greater, but these values ​​can be arbitrarily set according to desired conditions. Figure 9(b-3) shows the ΔHue values ​​of such a change (in this example, the threshold for the difference in hue is set to 60°). In Figure 9(b-3), the ΔHue indicating the change in hue is 70°, which is greater than or equal to the threshold, so it is determined that a change has occurred.

[0075] Furthermore, as in Embodiment 1, if multiple objects of the same category are detected, the hue variation may be calculated only for object regions detected at adjacent coordinates in the first and second generation results. Whether objects are detected at adjacent coordinates may be calculated using methods such as the Intersection over Union of the BBs (Block Blocks) of the detected object results.

[0076] In S3109, the auxiliary information generation unit 208 generates auxiliary information. In this modified example, since a change in color is detected as a changeable item, auxiliary information is generated to suppress the change in color. Here, the auxiliary information may include, for example, information prompting the user to add wording to specify the color to the prompt, or information prompting the user to change (modify) the prompt related to color. For example, in the example in Figure 9, since a change in the color of the frog was detected, auxiliary information such as, "A change in the color of 'frog' was detected in the generated images of model v2 and model v1. Please add wording to specify the color of the frog," may be generated.

[0077] <Effects of Variation 1-1> With this configuration, it is possible to detect color variations between the first and second generation results and generate auxiliary information to modify the input information to be input to the second model based on these variations. Therefore, even if the user uses a different model than the one previously used, it becomes easier to obtain a generation result with a similar color tone to the generation result of the previously used model.

[0078] <Variation 1-2> In Modification 1-1, the case where the object's color is used as a variable item was explained. In this modification, the case where the object's position and size are used as variable items will be explained. The object's position and size affect the composition of the image. Therefore, by prompting the server 103 to change the prompt in a way that suppresses fluctuations in the object's position and size, the user is more likely to obtain a generated image with less variation in composition. Here, only the points that differ from the explanation of Embodiment 1 described above will be explained.

[0079] In S3107, the detection unit 207, similar to the modified example 1-1, detects the variation between the second generation result obtained in S3105 and the first generation result obtained in S3106, and outputs it as a variation detection result.

[0080] The following explanation, similar to Modification 1-1, will describe an example of detecting changes in object size as a variable item, referring to Figure 9. First, the detection unit 207 uses an object detector, similar to Modification 1-1, to detect objects in the image from the first and second generation results, and outputs a list of object detection results as shown in Figure 9(b). In this example, the object region is detected by the object detector and output as the position and size of the BB region surrounding the object.

[0081] Next, the detection unit 207 calculates the average value of the object size (W) for each category for both the first and second generation results. t ',H t The results of ') are tallied. In the example in Figure 9(b), there is only one category of detected objects (frogs only), and in all generated results, only one object of that category is detected, so the calculation of the average value is unnecessary, (W t ',H t ')=(W t ,H t )

[0082] An example of the aggregated results calculated in this way is shown in Figure 9(e). Figure 9(e-1) shows that the average size of the frogs in the first generation result is (W1', H1'), and Figure 9(e-2) shows that the average size of the frogs in the second generation result is (W2', H2'). Based on these aggregated results, the detection unit 207 calculates the variation ΔS of the object size between the first generation result and the second generation result for each object category. ΔS can be calculated, for example, according to the following equation (2). The ΔS calculated by the following equation (2) represents the ratio of the object area between the first generation result and the second generation result. ΔS=(W2×H2) / (W1×H1) Equation (2)

[0083] In S3108, the detection unit 207 determines whether or not a change has occurred based on these aggregated results. Here, the detection unit 207 determines that a change has occurred if there are M or more object categories for each object category where ΔS is greater than or equal to the threshold. In the case of (W1', 1')=(120, 120) and (W2', H2')=(80, 80), ΔS for the object category "frog" is 2 / 3. Therefore, if the threshold is set to 0.5, it is determined that a change has occurred. Note that, as in Embodiment 1, if multiple objects of the same category are detected, the size change may be calculated only for object regions detected at adjacent coordinates in the first generation result and the second generation result.

[0084] In S3109, the auxiliary information generation unit 208 generates auxiliary information. In this modified example, since a change in size is detected as a change item, auxiliary information is generated to suppress the change in size. Here, the auxiliary information may include, for example, information prompting the user to add wording to specify the size to the prompt, or information prompting the user to change the size prompt. For example, in the example in Figure 9, since a change in the color of the frog was detected, auxiliary information such as, "A change in the size of the 'frog' was detected in the generated images of model v2 and model v1. Please add wording to specify the size of the frog," may be generated. Although an example in which a change in size is used as a change item has been described here, it is also possible to calculate a change in the position (X,Y) of the object as a change item, and if the change is greater than a threshold, prompt the user to add a prompt to suppress it.

[0085] <Effects of Modification 1-2> With this configuration, it is possible to detect changes in the position and size of an object between the first and second generation results, and to generate auxiliary information to modify the input information to be input to the second model based on these changes. Therefore, even if the user uses a different model than the one previously used, it is easier to obtain a generation result that has a similar position and size to the generation result of the previously used model, and with less variation in the image composition.

[0086] <Embodiment 2> The server 103 according to Embodiment 2 is capable of performing the same processing as the server 103 according to Embodiment 1. Furthermore, the server 103 according to Embodiment 2 evaluates the range of variation in the generation result when the input to the second model is changed from the first input information to the second input information, and displays information for changing the input information again if the range of variation is greater than a predetermined threshold.

[0087] The system including the server 103 according to this embodiment has the same hardware configuration as shown in Figure 1 as in Embodiment 1 and can perform the same processing, so redundant explanations will be omitted.

[0088] The server 103 according to this embodiment includes a storage unit 500, a model management unit 501, a control unit 502, an acquisition unit 503, a first result acquisition unit 504, a second result acquisition unit 505, a fluctuation range evaluation unit 506, a detection unit 507, an auxiliary information generation unit 508, an output unit 509, and a reception unit 510. Of these, the storage unit 500 is the same as the storage unit 200 in Embodiment 1, the model management unit 501 is the same as the model management unit 201, the acquisition unit 503 is the same as the acquisition unit 202, and the first result acquisition unit 504 is the same as the first result acquisition unit 203, so redundant explanations are omitted. Also, the second result acquisition unit 505 is the same as the second result acquisition unit 206, the auxiliary information generation unit 508 is the same as the auxiliary information generation unit 208, and the reception unit 510 is the same as the reception unit 209, so redundant explanations are omitted.

[0089] The following describes the functional units of the server 103 that differ from those in Embodiment 1. The control unit 502 controls the operation of the acquisition unit 503 and the detection unit 507 based on the model management information managed by the model management unit 501 and the fluctuation range evaluated by the fluctuation range evaluation unit 506.

[0090] The variation range evaluation unit 506 evaluates whether or not a variation has occurred between the two generation results. The evaluation by the variation range evaluation unit 506 to determine whether or not a variation has occurred between the two generation results can be performed in the same way as the process performed by the detection unit 507 in S3108 according to Embodiment 1. The two generation results used here may be a first generation result and a second generation result, or a first generation result and a third generation result.

[0091] The detection unit 507 detects the variation in the two generated results based on the variation range evaluated by the variation range evaluation unit 506.

[0092] The output unit 509 outputs the first generation result, the second generation result, and the third generation result as generation history information, and the storage unit 500 stores this output. The output unit 509 may also output the generation result or generation history information to a display device such as the display unit 101-7. The output unit 509 also outputs auxiliary information and presents it to the user. In this embodiment, the auxiliary information output in the same manner as in Embodiment 1 is referred to as "first auxiliary information" to distinguish it from the second auxiliary information described later. The output unit 509 can also generate second auxiliary information that prompts the user to re-modify the second input information based on the variation range evaluated by the variation range evaluation unit 506 based on the first generation result and the third generation result.

[0093] Figure 6(a) is a flowchart showing the overall processing flow according to this embodiment. In this embodiment, as in Embodiment 1, the difference between the first generation result and the second generation result is calculated when the model is updated. In Embodiment 1, the first generation result was explained as being stored in the storage unit 500, but as the number of users or the number of generation history information increases, a huge amount of memory capacity is required for the storage process by the storage unit 500. From this perspective, in this embodiment, the memory capacity can be reduced by configuring the system to perform the generation process of the first generation result using the first model again when the model is updated.

[0094] In S601, the acquisition unit 503 acquires model management information for each model, similar to S3101.

[0095] In S602, the acquisition unit 503 checks whether there is a model that has been updated, similar to S3102. If the model has been updated, the process proceeds to S603; otherwise, the process returns to S601.

[0096] In S603, the acquisition unit 503 selects the model that has been updated as the processing target and acquires generation history information including the first input information.

[0097] In S604, the acquisition unit 503 acquires a first model, which is the model of the target model before it was updated. In S605, the first result acquisition unit 505 inputs the first input information acquired in S603 into the first model and acquires the first generation result.

[0098] In S606, the acquisition unit 503 acquires the model being updated as a second model. In S607, the second result acquisition unit 505 inputs the first input information acquired in S603 into the second model and acquires the second generation result.

[0099] In S608, the detection unit 507 detects the variation between the first generation result obtained in S605 and the second generation result obtained in S607, and outputs it as a variation detection result. The processing performed in S608 according to this embodiment will be explained with reference to Figure 8(b). Here, we will explain the case in which a variation in the color of objects constituting the image is detected as the variation item, but different items such as the number, size, or position of objects may be used as the variation item.

[0100] The procedure shown in Figure 8(b) will be explained. First, in S620, the variation range evaluation unit 506 evaluates the variation range between the two generated results. The variation range evaluation unit 506 can, for example, calculate the difference in hue ΔHue, similar to the modified example 1, and evaluate this ΔHue as the variation range. In the example in Figure 9(d-3), ΔHue is 70°.

[0101] Next, in S621, the fluctuation range evaluation unit 506 performs fluctuation detection according to the fluctuation range evaluated in S620, outputs the fluctuation detection result, and proceeds to processing in S609. Here, the fluctuation range evaluation unit 506 can determine that there is a fluctuation if the fluctuation range is greater than a predetermined threshold, similar to the processing by the detection unit 207 in S3107 of Embodiment 1.

[0102] In S609, the fluctuation range evaluation unit 506 determines whether or not a fluctuation has occurred based on the fluctuation detection result obtained in S608. If a fluctuation has occurred, the process proceeds to S610; otherwise, the process in Figure 6(a) ends. The process in S609 can be executed in the same way as S3108 in Embodiment 1.

[0103] In S610, the auxiliary information generation unit 508 generates first auxiliary information in the same manner as in S3109. For example, if the color of the generated bear changes, the first auxiliary information may be generated in the form of a fill-in-the-blank sentence prompting a change in the prompt, such as "XX-colored bear," and the user may be prompted to input XX.

[0104] In S611, the output unit 509 displays the first auxiliary information on the display device, similar to S3110.

[0105] In S612, the reception unit 510 accepts user input changes and obtains second input information, similar to S3111. Here, in S610, first auxiliary information in the form of a fill-in-the-blank phrase such as "XX-colored bear" is generated, and input to fill in XX can be obtained from the user.

[0106] In S613, the second result acquisition unit 505 inputs the second input information into the second model, similar to S3112, and acquires the third generation result.

[0107] In S614, the detection unit 507 uses the first generation result and the third generation result to detect fluctuations in the same procedure as in S608 and outputs the fluctuation detection result.

[0108] In S615, the fluctuation range evaluation unit 506 determines whether or not a fluctuation has occurred based on the fluctuation detection result obtained in S614. If a fluctuation has occurred, the process proceeds to S616; otherwise, the process shown in Figure 6(a) ends. The process in S615 can be executed in the same way as in S609, except that the third generation result is used instead of the second generation result as the generation result used. Furthermore, the output unit 509 may present the third generation result to the user, and the storage unit 500 may store the third generation result linked to the generation history information corresponding to the first generation result. Here, the input information and model information used to generate the third generation result are also stored as generation history information.

[0109] In S616, the auxiliary information generation unit 508 generates second auxiliary information that prompts the user to change and re-enter the second input information. Here, the second auxiliary information is generated in the same way as the first auxiliary information and includes information for changing the input information to be entered into the second model from the second input information. The second auxiliary information is generated by the same process as the first auxiliary information, but it is a text prompt with different wording from the first auxiliary information.

[0110] Here, we consider the case where the auxiliary information generation unit 508 presents the user with a text prompt as first auxiliary information, "XX-colored bear," with a blank space of XX, and generates a prompt to be used as second input information by obtaining input from the user to fill in XX. In this case, the auxiliary information generation unit 508 can generate auxiliary information as second auxiliary information, which is auxiliary information for changing a variable item similar to the first auxiliary information, such as "a bear with a hue of XX" or "a bear with a XX-ish tint," with XX as the blank space, but with different wording from the first auxiliary information. Here, the auxiliary information generation unit 508 can generate as second auxiliary information the candidate that can be evaluated as being able to reduce the amount of variation the most when actually used as second auxiliary information, from among multiple candidates for second auxiliary information having the blank space described above (or multiple candidates in order of decreasing amount of variation). For example, the auxiliary information generation unit 508 can generate a candidate as second auxiliary information that is evaluated as having the smallest variation from the first generation result in the fourth generation result output by inputting the third input information (which has the blanks filled in using the blank content obtained in S612) used when generating the second input information into the second model. The text of such auxiliary information with blanks may be prepared in advance or estimated by an LLM or the like. For example, an image description generation model may be trained to estimate an editing instruction prompt to fill in the difference from two generated images (features) and use it as the text for the auxiliary information. The auxiliary information generation unit 508 may also generate blanks with adjectives such as "dark XX color" or "light XX color" added as auxiliary information, depending on the amount of variation. For example, among the candidates for second auxiliary information, only those whose variation when actually used as second auxiliary information is above a threshold may be recommended as second auxiliary information.

[0111] <Effects of Embodiment 2> In this way, the amount of variation is evaluated for each candidate of auxiliary information, and second auxiliary information can be generated in a way that is expected to be highly effective in suppressing variation (for example, in order of expected effectiveness). Therefore, even if the user uses a different model than the one previously used, it becomes easier to obtain generation results similar to those obtained with the previously used model. Furthermore, by generating and presenting auxiliary information to the user in order of expected effectiveness as described above, auxiliary information can be generated that makes it easier for the user to select a proposed change to the input.

[0112] <Embodiment 3> If the system were to perform a process to detect changes and generate auxiliary information for all generation history information, which consists of a set of prompt, model, and generation result, in conjunction with model updates, it could place a high load on the system. From this perspective, the server 103 according to this embodiment restricts the first generation result to be processed from among the first generation results of the model managed by the model management information, according to predetermined criteria. This reduces the load on the server even when there is a large amount of generation history information, and also reduces the effort required for users to change prompts. The server 103 according to Embodiment 3 can perform the same processing as the server 103 according to Embodiment 1, so redundant explanations will be omitted.

[0113] The system including the server 103 in this embodiment has the same hardware configuration as shown in Figure 1 and can perform the same processing as in Embodiment 1, so redundant explanations are omitted. Also, the server 103 in this embodiment has the same functional unit as shown in Figure 2 and can perform the same processing as in Embodiment 1, so redundant explanations are omitted.

[0114] As described above, the server 103 according to this embodiment restricts the models to be processed from among multiple models to those that meet predetermined criteria. Here, the predetermined criteria can be, for example, that the model has been pre-selected by the user (details will be described later), that the frequency of use is above a predetermined threshold, that fluctuations have been detected in the past, or that the magnitude of the fluctuation (fluctuation range) has exceeded a certain value. In the following explanation, it will be assumed that the predetermined criterion is that the model has been pre-selected by the user.

[0115] In this context, "frequency of use" refers to the frequency with which users perform generation processes using the model in question. This frequency may be the number of times generation processes are performed per predetermined period, or it may be the number of times users select to use the model per predetermined period.

[0116] Hereinafter, an example of information processing performed by the server 103 according to this embodiment will be described with reference to Figure 7. The processing shown in Figure 7 is just one example, and the server 103 does not need to perform all the processing described using this flowchart. The flowchart in Figure 7(a) shows the processing by the first model (first image generation processing (a)), and the flowchart in Figure 7(b) shows the processing by the second model (second image generation processing (a)).

[0117] [Processing by the first model] Figure 7(a) is a flowchart showing an example of the process when the first image generation process (a) is executed using the first model according to this embodiment. The process shown in Figure 7(a) is the same as that shown in Figure 3(a) of Embodiment 1, except that S7001 to S7002 are performed following S3005, so redundant explanations are omitted.

[0118] In S7001, the reception unit 209 accepts a pre-selection for the first model acquired in S3004. For example, in S7001, the output unit 204 may present the generation result or generation history information to the user using the display device 101-7, and the reception unit 209 may accept a pre-selection for the said generation result (or generation history information). The purpose of the pre-selection is to prioritize the change detection processing during model version upgrades and the generation processing of auxiliary information in subsequent processing using the selected generation history information.

[0119] In S7001, the specific processing performed is that the output unit 204 presents the generation result or generation history information to the user by displaying it on the display device 101-7. The user then checks the generation result or generation history information and decides whether to prioritize the change detection process, based on whether they want to generate images of a similar style again, and makes a pre-selection input. This decision method may involve, for example, a checkbox on the touch panel to specify whether to prioritize change detection, and the user's specification may be accepted as input to this checkbox. The generation result or generation history information presented here may be the first generation result or generation history information stored in S3005, or past generation results or generation history information may be displayed side by side on the touch panel, allowing the user to make multiple selections.

[0120] In S7002, the memory unit 200 stores information about a pre-selection flag, indicating whether it was pre-selected, in the generation history information, based on the pre-selection received in S7001. The value of the pre-selection flag is set to True if it was pre-selected, and False if it was not. Furthermore, if the user selects multiple generation history entries from past generation history information, the pre-selection flag is updated for each generation history entry.

[0121] Steps S3001 to S7002 shown in Figure 7(a) are performed each time an image is generated by the first model, and the storage unit 200 accumulates the generation history information. An example of the database of generation history information stored by the storage unit 200 is shown in Figure 10. In this example, one generation history information entry is stored in each row in a table format. Each column of each generation history information entry records the generation history information ID, the data of the generation result and the input (prompt) to the model used for its generation, the model type, the model version, a pre-selection flag indicating whether it has been pre-selected, and auxiliary information. For example, in generation history information ID 1, image 1 generated using prompt 1 with model 1 v1 (version 1) is stored, and S7002 indicates that it was pre-selected by the user. Other information such as the generation date and time may also be stored as generation history information, and the information stored here is not particularly limited. Furthermore, there are various types of models (e.g., model 1, model 2, model 3), and it is assumed that the user will select and use them, and that each will be updated with a new version.

[0122] Furthermore, while it is assumed here that the first generation result is stored as generation history information in S3005, the first generation result may be stored at the same time as the setting of the pre-selection flag information in S7002. Also, while it is assumed here that pre-selection is performed for each first generation result, for example, pre-selection may be accepted for the first model itself that is the target of processing in S3002 to S3005, and all generation results by that first model may be considered to have accepted pre-selection.

[0123] [Processing by the second model] In this process, when the first model is upgraded, the generation history information stored in the database that meets certain conditions is extracted. The conditions are that the first model is used, an older version is used, and it has been pre-selected. Then, the first prompt is input to the second model, which is the upgraded model, and the second generation result is output. Then, the system detects the difference between the first and second generation results, and if there is a difference, it generates auxiliary information. In the example in Figure 10, when the first model, Model 1, is upgraded to V3, the system detects the difference in generation history using older models, V1 and V2, which are older than V3.

[0124] Figure 7(b) is a flowchart showing an example of the process when the second image generation process (b) is executed using the second model according to this embodiment. The process shown in Figure 7(b) is performed in the same way as that shown in Figure 3(b) of Embodiment 1, except that S7011 to S7014 are performed instead of S3104, and the process from S3111 onwards is omitted, so redundant explanations are omitted.

[0125] In the process shown in Figure 7(b), after S3101 to S3103 in Figure 3(b) are executed, S7011 to S3110 are repeatedly executed for generation history information ID=1 to ID=N in the generation history information database stored in the storage unit 200. In the loop processing of S7011 to S3108, the detection unit 207 detects the change between the first generation result and the second generation result and outputs it as a change detection result. This change detection is performed using the first model and is limited to (limited to) the pre-selected generation history information.

[0126] In S7011, the detection unit 207 acquires one of the generation history information stored in the storage unit 200 as the target for processing. Here, the total number of generation history information to be acquired is N, and each generation history information is assigned an ID from 1 to N in the order in which it is acquired. In this loop processing from S7011 to S3110, in S7011, the generation history information is acquired as the target for processing in order from the smallest ID. Here, the detection unit 207 may reduce the processing load by setting an upper limit instead of acquiring all of the generation history information. In that case, the detection unit 207 may also consider narrowing down the generation history information to be acquired to those that the user is likely to reuse, and acquire them in order from the newest generation history information (from the newest associated generation date and time).

[0127] In S7012, the detection unit 207 determines whether the generation history information to be processed was generated by the first model. If the generation history information to be processed was generated by the first model, the process proceeds to S7013; otherwise, the process proceeds to S7011. For example, if the generation history information database shown in Figure 10 is used, and the first model is Model 1, then the generation history information for which the model is Model 1 will be extracted from the database.

[0128] In S7013, the detection unit 207 determines whether the model version associated with the generation history information of the target to be processed is the latest version. If the model version associated with the generation history information of the target to be processed is the latest version, the process proceeds to S7014; otherwise, the process returns to S7011. In this example, the process proceeds to S7014 if the model version associated with the generation history information is not V3.

[0129] In S7014, the detection unit 207 determines whether the value of the pre-selection flag associated with the generation history information to be processed is True (i.e., it has been pre-selected). If the value of the pre-selection flag associated with the generation history information to be processed is True, the process proceeds to S3105; otherwise, the process returns to S7011.

[0130] Next, the generation history information acquired in S7011 is processed in the same way as in S3105 to S3110 in Embodiment 1. If no change is detected in S3108, the process returns to S7011. If a change is detected in S3108, auxiliary information is generated in S3109 to S3110, recorded in the generation history information database, and then the process returns to S7011. The auxiliary information also records which version it was generated for. In Figure 10, for generation history information ID 1, auxiliary information is generated and recorded because Model 1 was upgraded from V1 to V3.

[0131] The processing described in Figure 7(b) up to this point is performed when the model is updated. The third generation result output processing using the auxiliary information from S3111 to S3113 in Embodiment 1 is performed at the user's discretion.

[0132] <Effects of Embodiment 3> With this configuration, by limiting the scope of change detection within the generation history information, the load on the server can be reduced even when there is a large amount of generation history information. Furthermore, it reduces the effort required for users to change prompts.

[0133] The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) A first acquisition means for inputting first input information into a first machine learning model and obtaining a first generation result, A second acquisition means for inputting the first input information into a second machine learning model different from the first machine learning model to obtain a second generation result, A detection means for detecting a first variation in an item that varies according to the difference in the machine learning model used for generation between the first generation result and the second generation result, A first generation means generates first auxiliary information, which includes information for changing the input information to be input to the second machine learning model from the first input information, based on the first variation when the first variation is detected. An information processing device equipped with the following features. (Item 2) The information processing apparatus according to item 1, wherein the second acquisition means is characterized by inputting a second input information different from the first input information into the second machine learning model to further acquire a third generation result. (Item 3) Presentation means for presenting the first auxiliary information described above to the user, A third acquisition means presents the first auxiliary information to the user and acquires user input, A second generation means that generates second input information by modifying the first input information based on the user input, The information processing device according to item 1 or 2, further comprising: (Item 4) The information processing apparatus according to item 1, characterized in that the first generation means generates the first auxiliary information such that the second variation of the item between the first generation result and the third generation result obtained by inputting the second input information, which is the first input information modified based on the first auxiliary information, into the second machine learning model, is smaller than the first variation. (Item 5) The information processing device according to item 4, characterized in that the first auxiliary information is text information that promotes the suppression of the first fluctuation. (Item 6) The first input information includes a text prompt, The information processing device according to item 5, characterized in that the first auxiliary information is information prompting correction of the text prompt. (Item 7) The system further includes an evaluation means for evaluating the range of the first variation, The information processing device according to any one of items 1 to 6, wherein the detection means detects the first variation based on an evaluation of the variation range of the first variation. (Item 8) The evaluation means further evaluates the range of variation of the second variation of the item between the first generation result and the third generation result obtained by inputting a second input information different from the first input information into the second machine learning model, The detection means further detects the second variation based on an evaluation of the variation range of the second variation. The information processing device according to item 7, characterized in that the first generation means further generates second auxiliary information which includes information for changing the input information to be input to the second machine learning model from the second input information when the second variation is detected. (Item 9) The first auxiliary information described above is text information that encourages the suppression of the first fluctuation, The information processing device according to item 8, characterized in that the second auxiliary information is text information different from the first auxiliary information that encourages the suppression of the first fluctuation. (Item 10) The first generation means generates a plurality of the second auxiliary information, The evaluation means further evaluates the range of variation of the third item between the second generation result and the fourth generation result obtained when the third input information, which is obtained by modifying the second input information based on the second auxiliary information, is input to the second machine learning model, for each of the second auxiliary information. The information processing device according to item 9, characterized in that the multiple second auxiliary pieces of information are presented in an order based on the variation range of the third variation. (Item 11) The first acquisition means acquires the first generation result from each of the multiple machine learning models, The information processing device according to any one of items 1 to 10, characterized in that the detection means detects a first variation of the item between the first generation result and the second generation result only for the first generation result that satisfies predetermined conditions. (Item 12) The information processing device according to item 11, characterized in that the aforementioned predetermined conditions are selected in advance by the user. (Item 13) The information processing apparatus according to item 11, characterized in that the predetermined condition is that the frequency with which the machine learning model that generates the corresponding first generation result is used by the user is above a predetermined threshold. (Item 14) The information processing apparatus according to item 11, characterized in that the predetermined condition is that the first generation result is one in which a first variation of an item between it and the second generation result has been detected by the detection means, or that the magnitude of the first variation of an item between it and the second generation result has been greater than or equal to a predetermined threshold. (Item 15) Furthermore, it includes a management mechanism for managing information indicating whether or not a user can use one of several machine learning models. The information processing device according to any one of items 1 to 14, characterized in that the first machine learning model is a machine learning model among the plurality of machine learning models that is permitted for use by the management means to the user. (Item 16) The first generation result and the second generation result are image data. The information processing device according to any one of items 1 to 15, characterized in that the item is the number of specific objects, the color of the specific objects, the position of the specific objects, or the size of the specific objects in the first generation result and the second generation result. (Item 17) The process involves inputting the first input information into the first machine learning model and obtaining the first generated result, A step of inputting the first input information into a second machine learning model different from the first machine learning model to obtain a second generation result, A step of detecting a first variation in an item that varies depending on the difference in the machine learning model used for generation between the first generation result and the second generation result, A step of generating first auxiliary information, which includes information for changing the input information to be input to the second machine learning model from the first input information, when the first variation is detected, based on the first variation; An information processing method comprising the following: (Item 18) A program to cause a computer to function as one of the means of an information processing device described in any one of items 1 through 16.

[0134] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0135] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0136] 101: Client, 102: Network, 103: Server

Claims

1. A first acquisition means for inputting first input information into a first machine learning model and obtaining a first generation result, A second acquisition means for inputting the first input information into a second machine learning model different from the first machine learning model to obtain a second generation result, A detection means for detecting a first variation in an item that varies according to the difference in the machine learning model used for generation between the first generation result and the second generation result, A first generation means generates first auxiliary information, which includes information for changing the input information to be input to the second machine learning model from the first input information, based on the first variation when the first variation is detected. An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, characterized in that the second acquisition means inputs a second input information different from the first input information into the second machine learning model to further acquire a third generation result.

3. The first means for presenting the auxiliary information described above to the user, A third acquisition means presents the first auxiliary information to the user and acquires user input, A second generation means that generates second input information by modifying the first input information based on the user input, The information processing apparatus according to claim 1, further comprising:

4. The information processing apparatus according to claim 1, characterized in that the first generation means generates the first auxiliary information such that the second variation of the item between the first generation result and the third generation result obtained by inputting the second input information, which is the first input information modified based on the first auxiliary information, into the second machine learning model and obtaining it, is smaller than the first variation.

5. The information processing apparatus according to claim 4, characterized in that the first auxiliary information is text information that promotes the suppression of the first fluctuation.

6. The first input information includes a text prompt, The information processing apparatus according to claim 5, characterized in that the first auxiliary information is information prompting correction of the text prompt.

7. The system further includes an evaluation means for evaluating the range of the first variation, The information processing apparatus according to claim 1, characterized in that the detection means detects the first variation based on an evaluation of the variation range of the first variation.

8. The evaluation means further evaluates the range of variation of the second variation of the item between the first generation result and the third generation result obtained by inputting a second input information different from the first input information into the second machine learning model, The detection means further detects the second variation based on an evaluation of the variation range of the second variation. The information processing apparatus according to claim 7, characterized in that the first generation means further generates second auxiliary information including information for changing the input information to be input to the second machine learning model from the second input information when the second variation is detected.

9. The first auxiliary information is text information that encourages the suppression of the first fluctuation, The information processing apparatus according to claim 8, characterized in that the second auxiliary information is text information different from the first auxiliary information that encourages the suppression of the first fluctuation.

10. The first generation means generates a plurality of the second auxiliary information, The evaluation means further evaluates the range of variation of the third item between the second generation result and the fourth generation result obtained when the third input information, which is obtained by modifying the second input information based on the second auxiliary information, is input to the second machine learning model, for each of the second auxiliary information. The information processing apparatus according to claim 9, characterized in that the plurality of second auxiliary pieces of information are presented in an order based on the variation range of the third variation.

11. The first acquisition means acquires the first generation result from each of the multiple machine learning models, The information processing apparatus according to claim 1, characterized in that the detection means detects a first variation of the item between the first generation result and the second generation result only for the first generation result that satisfies predetermined conditions.

12. The information processing apparatus according to claim 11, characterized in that the aforementioned predetermined conditions are selected in advance by the user.

13. The information processing apparatus according to claim 11, characterized in that the predetermined condition is that the frequency with which the machine learning model that generates the corresponding first generation result is used by the user is above a predetermined threshold.

14. The information processing apparatus according to claim 11, characterized in that the predetermined condition is a first generation result in which a first variation of an item between it and the second generation result has been detected by the detection means, or a first generation result in which the magnitude of the first variation of an item between it and the second generation result has been greater than or equal to a predetermined threshold.

15. Furthermore, it includes a management mechanism for managing information indicating whether or not a user can use one of several machine learning models. The information processing apparatus according to claim 1, characterized in that the first machine learning model is a machine learning model among the plurality of machine learning models that has been permitted for use by the management means to the user.

16. The first generation result and the second generation result are image data, The information processing apparatus according to claim 1, characterized in that the item is the number of specific objects, the color of the specific objects, the position of the specific objects, or the size of the specific objects in the first generation result and the second generation result.

17. A process of inputting the first input information into the first machine learning model and obtaining the first generated result, A step of inputting the first input information into a second machine learning model different from the first machine learning model to obtain a second generation result, A step of detecting a first variation in an item that varies depending on the difference in the machine learning model used for generation between the first generation result and the second generation result, A step of generating first auxiliary information, which includes information for changing the input information to be input to the second machine learning model from the first input information, when the first variation is detected, based on the first variation, An information processing method comprising the following:

18. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 16.

Images