Information processing device, information processing method, and information processing program

Abstract

To generate an image that intuitively expresses the state of air. [Solution] An information processing device having a control unit, wherein the control unit acquires environmental data of a target environment and converts it into natural language including sensory expressions to generate environmental language, generates prompts including words or sentences that express images that co-occur from the environmental language, selects a generation AI that has previously undergone additional training using multiple learning data based on the generated prompt, operates it based on the assigned weights, and displays an image showing the air condition of the target environment generated by the generation AI.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] There is known a technique for visualizing environmental data such as temperature, humidity, wind speed, and carbon dioxide concentration. This technique makes it possible to quantitatively grasp, for example, the state of the air, such as temperature, humidity, wind speed, and carbon dioxide concentration. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-60907 Summary of the Invention [Problem to be solved by the invention]

[0004] However, until now, there has been no way to intuitively express air quality (for example, to visualize human sensations). For example, even using generative AI, it has been difficult to directly generate an image that intuitively expresses air quality from quantitative data such as those described above.

[0005] The present disclosure aims to generate an image that intuitively expresses the state of air. [Means for solving the problem]

[0006] A first aspect of the present disclosure is an information processing device having a control unit, The control unit Environmental data of the target environment is acquired and converted into natural language including sensory expressions to generate environmental language. generating prompts including phrases or sentences that represent images that co-occur from the ambient language; Based on the generated prompt, a generation AI that has been trained in advance using multiple learning data sets is selected, and it operates based on the assigned weights. An image showing the air condition of the target environment generated by the generation AI is displayed.

[0007] A second aspect of the present disclosure is the information processing device according to the first aspect, The generation AI uses a model for generating a specific image and is additionally trained using corresponding training data, The corresponding learning data includes a group of specific images and captions indicating each component included in each specific image.

[0008] A third aspect of the present disclosure is the information processing device according to the second aspect, The learning data corresponding to the model for generating the specific image is associated with a score for each component based on the reliability of each component calculated when recognizing each component contained in each specific image.

[0009] A fourth aspect of the present disclosure is the information processing device according to any one of the first to third aspects, The control unit Each component is extracted from the generated prompt, and the total score for each learning data is calculated by calculating the sum of the scores for each extracted component for each learning data based on the scores for each component associated with the learning data.

[0010] A fifth aspect of the present disclosure is the information processing device according to the fourth aspect, The control unit Selecting training data based on the total score; When operating the generation AI that has been additionally trained using a model corresponding to the selected learning data, the weight of the corresponding model is calculated based on the total score.

[0011] A sixth aspect of the present disclosure is the information processing device according to the fourth aspect, The control unit Selecting training data based on the total score; Using a large-scale language model, determine the importance of the model corresponding to the selected training data for the prompt; Based on the determined importance, the weight of the model is calculated when operating the generation AI that has been additionally trained using a model corresponding to the selected learning data.

[0012] A seventh aspect of the present disclosure is the information processing device according to the fourth aspect, The control unit Selecting training data based on the total score; The environmental conditions assigned to the selected learning data are determined.

[0013] An eighth aspect of the present disclosure is the information processing device according to the seventh aspect, The control unit If an environmental condition is assigned to the selected training data and the sentence included in the prompt satisfies the environmental condition, a generation AI that has been additionally trained using a model corresponding to the selected training data is operated based on a predetermined weight; If an environmental condition is assigned to the selected training data and the sentence included in the prompt does not satisfy the environmental condition, a generation AI that has been additionally trained using a model other than the model corresponding to the selected training data is operated based on a predetermined weight; If the selected learning data does not have any environmental conditions attached to it, a generation AI that has been additionally trained using a model corresponding to any learning data is operated based on predetermined weights.

[0014] A ninth aspect of the present disclosure is the information processing device according to the fourth aspect, The control unit Selecting training data based on the total score; Obtaining a plurality of combinations of weights for the model when operating the generation AI that has been additionally trained using a model corresponding to the selected training data; The generation AI, which has been additionally trained using a model corresponding to the selected learning data, is operated based on each weight combination, and the weight combination corresponding to the image evaluated by the user is identified from among the images generated by the generation AI.

[0015] A tenth aspect of the present disclosure is an information processing device according to any one of the first to ninth aspects, The learning data is First learning data for additionally learning the image configuration; Second learning data for additionally learning predetermined elements contained in the image; Third learning data for additional learning of the environment represented by the image; Including, The control unit A generation AI that has been additionally trained using the first training data is operated to obtain a first image generated by the generation AI; A generation AI that has been additionally trained using the second training data is operated using the first image to obtain a second image generated by the generation AI; A generation AI that has been additionally trained using the third learning data is operated using the second image to obtain a third image generated by the generation AI; The acquired third image is displayed as an image showing the air condition of the target environment.

[0016] An eleventh aspect of the present disclosure is an information processing device having a control unit, The control unit Recognizing each component included in each specific image; Using learning data including a group of specific images and captions indicating each component recognized in each specific image, additional training of the generation AI is performed using a model for generating specific images; A score for each component element according to the reliability of each component element calculated when recognizing each component element included in each specific image is stored in association with the learning data.

[0017] A twelfth aspect of the present disclosure is the information processing device according to the eleventh aspect, The learning data is First learning data for additionally learning the image configuration; Second learning data for additionally learning predetermined elements contained in the image; and third learning data for additionally learning the environment represented by the image.

[0018] A thirteenth aspect of the present disclosure is an information processing method, A control unit included in the information processing device Environmental data of the target environment is acquired and converted into natural language including sensory expressions to generate environmental language. generating prompts including phrases or sentences that represent images that co-occur from the ambient language; Based on the generated prompt, a generation AI that has been trained in advance using multiple learning data sets is selected, and it operates based on the assigned weights. A process is executed to display an image showing the air condition of the target environment generated by the generation AI.

[0019] A fourteenth aspect of the present disclosure is an information processing method, A control unit included in the information processing device Recognizing each component included in each specific image; Using learning data including a group of specific images and captions indicating each component included in each specific image, additional training is performed on the generation AI using a model for generating specific images, A process is executed in which a score for each component, calculated when recognizing each component included in each specific image and corresponding to the reliability of each component, is stored in association with the learning data.

[0020] A fifteenth aspect of the present disclosure is an information processing program, A control unit of the information processing device includes: Environmental data of the target environment is acquired and converted into natural language including sensory expressions to generate environmental language. generating prompts including phrases or sentences that represent images that co-occur from the ambient language; Based on the generated prompt, a generation AI that has been trained in advance using multiple learning data sets is selected, and it operates based on the assigned weights. A process is executed to display an image showing the air condition of the target environment generated by the generation AI.

[0021] A sixteenth aspect of the present disclosure is an information processing program, A control unit of the information processing device includes: Recognizing each component included in each specific image; Using learning data including a group of specific images and captions indicating each component included in each specific image, additional training is performed on the generation AI using a model for generating specific images, A process is executed in which scores for each component, calculated when recognizing each component included in each specific image according to the reliability of each component, are stored in association with the learning data. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a first diagram showing an example of a system configuration of an image generation system. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 3] FIG. 1 is a first diagram illustrating an example of a functional configuration of an information processing device. [Figure 4] 10A and 10B are diagrams illustrating a specific example of processing by an environmental data acquisition unit. [Figure 5] FIG. 10 is a first diagram showing a specific example of processing by the environment language generation unit. [Figure 6] FIG. 2 is a second diagram showing a specific example of processing by the environment language generation unit. [Figure 7] FIG. 3 is a third diagram showing a specific example of processing by the environment language generation unit. [Figure 8] FIG. 10 is a first diagram showing a specific example of processing by a prompt generation unit. [Figure 9] FIG. 10 is a second diagram showing a specific example of processing by the prompt generation unit. [Figure 10] FIG. 10 is a third diagram showing a specific example of processing by the prompt generation unit. [Figure 11] FIG. 4 is a fourth diagram showing a specific example of processing by the prompt generation unit. [Figure 12] FIG. 5 is a fifth diagram showing a specific example of processing by the prompt generation unit. [Figure 13] FIG. 6 is a sixth diagram showing a specific example of processing by the prompt generation unit. [Figure 14] FIG. 7 is a seventh diagram showing a specific example of processing by the prompt generation unit. [Figure 15] FIG. 8 is an eighth diagram showing a specific example of processing by the prompt generation unit. [Figure 16] FIG. 9 is a ninth diagram showing a specific example of processing by the prompt generation unit. [Figure 17] 1 is a first flowchart showing the flow of image generation processing by the image generation system. [Figure 18] FIG. 1 is a first diagram showing an example of a display screen. [Figure 19] FIG. 2 is a second diagram illustrating an example of the functional configuration of the information processing device. [Figure 20] FIG. 2 is a second diagram showing an example of a display screen. [Figure 21] FIG. 3 is a third diagram illustrating an example of a functional configuration of an information processing device. [Figure 22] 10A and 10B are diagrams illustrating a specific example of processing by a motion control language generation unit. [Figure 23] 1A and 1B are diagrams showing an example of a prompt (sentence) and an example of a moving image. [Figure 24] 10 is a second flowchart showing the flow of image generation processing by the image generation system. [Figure 25]FIG. 2 is a second diagram showing an example of the system configuration of the image generation system. [Figure 26] FIG. 4 is a fourth diagram illustrating an example of the functional configuration of the information processing device. [Figure 27] 10A and 10B are diagrams illustrating a specific example of processing by a caption generating unit. [Figure 28] FIG. 10 is a diagram illustrating a specific example of processing by a learning data generation unit. [Figure 29] FIG. 10 is a diagram showing details of a generation AI and a fine tuning unit. [Figure 30] 10 is a flowchart showing the flow of additional learning processing by the image generation system. [Figure 31] FIG. 3 is a third diagram showing an example of the system configuration of the image generation system. [Figure 32] FIG. 5 is a fifth diagram illustrating an example of the functional configuration of an information processing device. [Figure 33] FIG. 10 is a first diagram showing a specific example of processing by a weight calculation unit. [Figure 34] 10 is a third flowchart showing the flow of image generation processing by the image generation system. [Figure 35] FIG. 10 is a second diagram showing a specific example of processing by the weight calculation unit. [Figure 36] FIG. 6 is a sixth diagram illustrating an example of the functional configuration of the information processing device. [Figure 37] 10A and 10B are diagrams illustrating a specific example of processing by a selection unit. [Figure 38] FIG. 7 is a seventh diagram illustrating an example of the functional configuration of the information processing device. [Figure 39] FIG. 10 is a third diagram showing a specific example of processing by the weight calculation unit. [Figure 40] 10 is a fourth flowchart showing the flow of image generation processing by the image generation system. [Figure 41] FIG. 4 is a fourth diagram showing an example of the system configuration of the image generation system. [Figure 42] FIG. 10 is a diagram showing specific examples of learning data for each category. [Figure 43] FIG. 5 is a fifth diagram showing an example of the system configuration of the image generation system. [Figure 44] 10 is a fifth flowchart showing the flow of image generation processing by the image generation system. [Figure 45] 1A and 1B are diagrams illustrating an example of an image generation process performed by an image generation system. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0024] [First embodiment] <System configuration of image generation system> A description will be given of the system configuration of an image generation system to which an information processing device according to the first embodiment is applied. Fig. 1 is a first diagram showing an example of the system configuration of the image generation system.

[0025] 1, the image generation system 100 includes a server device 110, an information processing device 120, and a server device 140. In the image generation system 100, the information processing device 120, the server device 110, and the server device 140 are communicably connected via a network 150.

[0026] The server device 110 has a generation AI 111, and when it receives a prompt from the information processing device 120 via the network 150, it operates the generation AI 111 to generate an image (a still image or a moving image) according to the prompt. The server device 110 transmits the generated image to the information processing device 120 via the network 150.

[0027] The information processing device 120 is a device that displays an image that intuitively expresses the state of air in a target environment. In the first embodiment, the information processing device 120 causes the generation AI 111 to generate an image (still image) that intuitively expresses the state of air in the target environment, and acquires and displays the image from the generation AI 111. Note that the target environment refers to a space that is the target when expressing the state of air. The information processing device 120 executes various processes required for the generation AI 111 to generate an image.

[0028] Specifically, the information processing device 120: Environmental data (such as temperature, humidity, etc.) of the target environment measured by sensors 1 to n (reference numerals 130_1 to 130_n) installed in the target environment, or Environmental data of the target environment entered by the user (e.g., data on season, location, etc.); The information processing device 120 acquires environmental data of the target environment (for example, data on rainfall, wind speed, etc.) from the server device 140 via the network 150.

[0029] The information processing device 120 generates environmental language by converting the acquired environmental data of the target environment into natural language including sensory expressions. The information processing device 120 generates prompts including words, sentences, or images that express images that co-occur from either or both of the environmental data and the environmental language. The information processing device 120 operates a generation AI using the generated prompts, and displays an image generated by the generation AI that indicates the air condition of the target environment.

[0030] The server device 140 functions as an information providing unit 141 and provides environmental data of the target environment (for example, data on rainfall, wind speed, etc.) to the information processing device 120 via the network 150. For example, the server device 140 acquires location information (latitude, longitude, altitude) of the target environment from the information processing device 120, and collects environmental data corresponding to the location information at predetermined time intervals. The server device 140 provides the collected environmental data to the information processing device 120 at predetermined time intervals.

[0031] 1 illustrates a case where the information processing device 120 is installed outside the target environment, but the information processing device 120 may be installed within the target environment. Also, the example of Fig. 1 does not mention a user viewing an image showing the air condition of the target environment displayed by the information processing device 120, but the user may view the image either within the target environment or outside the target environment.

[0032] <Hardware configuration of information processing device> The hardware configuration of the information processing device 120 will be described. Fig. 2 is a diagram showing an example of the hardware configuration of the information processing device. As shown in Fig. 2, the information processing device 120 has a processor 201, a memory 202, an auxiliary storage device 203, a user interface device 204, a communication device 205, a connection device 206, and a drive device 207. The hardware components of the information processing device 120 are connected to each other via a bus 208.

[0033] The processor 201 has various arithmetic devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 201 reads various programs (for example, information processing programs, etc.) into the memory 202 and executes them.

[0034] The memory 202 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 201 and the memory 202 form a so-called computer (also referred to as a "control unit 200"), and the processor 201 executes various programs read onto the memory 202, causing the computer to realize various functions.

[0035] The auxiliary storage device 203 stores various programs and various information used when the processor 201 executes the various programs.

[0036] The user interface device 204 includes an operation device for inputting user instructions and a display device for displaying a display screen including images.

[0037] The communication device 205 is a device that is connected to the network 150 and performs communication processing with the server devices 110, 140, and the like.

[0038] The connection device 206 is a device that connects the sensors 1 to n (reference numerals 130_1 to 130_n) and the information processing device 120.

[0039] The drive device 207 is a device for loading a recording medium 210. The recording medium 210 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The recording medium 210 may also include semiconductor memory that records information electrically, such as a ROM, a flash memory, etc.

[0040] The various programs to be installed in the auxiliary storage device 203 are installed, for example, by setting the distributed recording medium 210 in the drive device 207 and reading the various programs recorded on the recording medium 210 by the drive device 207. Alternatively, the various programs to be installed in the auxiliary storage device 203 may be installed by being downloaded from the network 150 via the communication device 205.

[0041] <Functional configuration of information processing device> The functional configuration of the information processing device 120 will be described. FIG. 3 is a first diagram showing an example of the functional configuration of the information processing device. As described above, an information processing program is installed in the information processing device 120, and by executing the information processing program, the information processing device 120: A communication control unit 310, an environmental data acquisition unit 320; ·Environmental language generation unit 330, a prompt generator 340; an output unit 350; It functions as:

[0042] Communication control unit 310 acquires environmental data from server device 140 via network 150 and notifies environmental data acquisition unit 320. Communication control unit 310 acquires prompts generated by prompt generation unit 340 and transmits them to server device 110 via network 150, and also receives images from server device 110 and notifies output unit 350.

[0043] The environmental data acquisition unit 320 Environmental data notified by the communication control unit 310, Environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), Environmental data entered by the user; and notifies the environment language generation unit 330 and the prompt generation unit 340.

[0044] The environmental language generation unit 330 generates environmental language based on the environmental data notified by the environmental data acquisition unit 320 by referencing the conversion table stored in the conversion table storage unit 360. Environmental language is natural language that intuitively expresses the state of the air in the target environment. "Sensuous expression" refers to putting into words the sensations that a person actually in the target environment would feel. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0045] The prompt generator 340 A phrase, sentence, or image expressing an image that co-occurs from either or both of the environmental data notified from the environmental data acquisition unit 320 and the environmental language notified from the environmental language generation unit 330; Image editing parameters according to the environmental data notified from the environmental data acquisition unit 320; Prompt generating section 340 notifies communication control section 310 of the generated prompt, and also notifies output section 350. Note that the image editing parameters refer to quantitative indices that affect image quality, and specific examples will be described later.

[0046] The output unit 350 generates a display screen based on the image notified by the communication control unit 310 and the wording or sentence, image editing parameters, etc. contained in the prompt notified by the prompt generation unit 340, and displays it to the user.

[0047] <Specific examples of processing by each unit of the information processing device> A specific example of processing by each unit (here, the environment data acquisition unit 320, the environment language generation unit 330, and the prompt generation unit 340) of the information processing device 120 will be described.

[0048] (1) Specific examples of processing by the environmental data acquisition unit Fig. 4 is a diagram showing a specific example of processing by the environmental data acquisition unit. In Fig. 4, reference numeral 401 indicates an example of environmental data provided by the server device 140 and acquired by the environmental data acquisition unit 320 from the communication control unit 310. As indicated by reference numeral 401 in Fig. 4, the environmental data provided by the server device 140 includes rainfall, wind direction, wind speed, weather, a discomfort index, etc.

[0049] 4, reference numeral 402 indicates an example of environmental data acquired by the environmental data acquisition unit 320 through measurements made by sensors 1 to n (reference numerals 130_1 to 130_n) or input by a user. As indicated by reference numeral 402 in Fig. 4, the environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n) or input by a user includes temperature, humidity, date, time, carbon dioxide concentration, season, location, etc.

[0050] The environmental data acquisition unit 320 notifies the acquired environmental data to the environmental language generation unit 330. When notifying the environmental data, the environmental data acquisition unit 320 may separate each item included in the environmental data into an item indicating the air condition of the target environment and an item that affects the air condition of the target environment.

[0051] The example of Fig. 4 shows how temperature, humidity, discomfort index, rainfall, wind direction, wind speed, carbon dioxide concentration, etc. are notified to the environmental language generation unit 330 as items indicating the air condition of the target environment. The example of Fig. 4 also shows how date, time, season, weather, location, etc. are notified to the environmental language generation unit 330 as items that affect the air condition of the target environment. As shown in Fig. 4, the item values ​​of each item indicating the air condition of the target environment and the item values ​​of each item that affects the air condition of the target environment may be notified to the environmental language generation unit 330 at predetermined time intervals, for example.

[0052] (2) Specific examples of processing by the environment language generation unit Fig. 5 is a first diagram showing a specific example of processing by the environmental language generation unit. As shown in Fig. 5, the conversion table storage unit 360 stores a conversion table in which item values ​​of multiple items indicating the air condition of the target environment among the environmental data are associated with environmental language, which is natural language including sensory expressions.

[0053] Of these, conversion table 510 is a table in which each item value of the item "temperature" is associated with an environmental language. Conversion table 520 is a table in which each item value of the item "humidity" is associated with an environmental language. Conversion table 530 is a table in which each item value of the item "discomfort index" is associated with an environmental language. Conversion table 540 is a table in which each item value of the item "rainfall" is associated with an environmental language. Conversion table 550 is a table in which each item value of the item "wind speed" is associated with an environmental language. When each item value of an item indicating the air condition of the target environment is included as in conversion tables 510 to 550, By logarithmically transforming each item value, or By approximating each item value with a sigmoid function, A conversion table that takes into consideration compatibility with bodily sensations may also be used.

[0054] When the environmental language generation unit 330 receives environmental data from the environmental data acquisition unit 320, such as item values ​​of items indicating the air condition of the target environment as temperature, humidity, discomfort index, rainfall, wind speed, etc., it refers to the conversion tables 510-550. The environmental language generation unit 330 converts the item values ​​of items indicating the air condition of the target environment as temperature, humidity, discomfort index, rainfall, wind speed, etc. into natural language including sensory expressions to generate environmental language. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0055] Fig. 6 is a second diagram showing a specific example of processing by the environmental language generation unit. As shown in Fig. 6, the conversion table storage unit 360 stores a conversion table in which item values ​​of multiple items that affect the air condition of the target environment are associated with environmental languages, which are natural languages ​​including sensory expressions.

[0056] Of these, the conversion table 610 is a table in which each item value of the item="date" is associated with the environment language.

[0057] When the environmental language generation unit 330 receives environmental data from the environmental data acquisition unit 320, such as the date, as an item value that affects the air quality of the target environment, the environmental language generation unit 330 refers to the conversion table 610. The environmental language generation unit 330 converts the date, such an item value that affects the air quality of the target environment, into natural language that includes sensory expressions, and generates an environmental language. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0058] Fig. 7 is a third diagram showing a specific example of processing by the environmental language generation unit. As shown in Fig. 7, the conversion table storage unit 360 stores a conversion table in which combinations of item values ​​of multiple items that affect the air condition of the target environment are associated with environmental languages, which are natural languages ​​including sensory expressions.

[0059] Of these, conversion table 710 is a table in which combinations of item values ​​for items="season", "date", and "location" are associated with environmental languages. Similarly, conversion table 720 is a table in which combinations of item values ​​for items="season", "date", and "location" are associated with environmental languages. Conversion table 710 and conversion table 720 have the same items but different item values. Therefore, the associated environmental languages ​​are different.

[0060] When the environmental language generation unit 330 receives environmental data from the environmental data acquisition unit 320, such as a combination of item values ​​of items that affect the air quality of the target environment, such as season, date, and location, the environmental language generation unit 330 refers to the conversion tables 710-720. The environmental language generation unit 330 converts the combination of item values ​​of items that affect the air quality of the target environment, such as season, date, and location, into natural language that includes sensory expressions, and generates an environmental language. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0061] (3) Example of processing by the prompt generator 8 is a first diagram showing a specific example of processing by the prompt generation unit. As described above, prompt generation unit 340 generates a prompt including a sentence expressing an image that co-occurs from the environmental data notified from environmental data acquisition unit 320 (here, the item values ​​of the items that affect the air quality of the target environment), and notifies communication control unit 310 of the generated prompt.

[0062] 8 shows how the prompt generation unit 340 generates sentences expressing images that co-occur with the respective item values ​​when the communication control unit 310 notifies the communication control unit 310 of the item values ​​that affect the air quality of the target environment. The prompt generation unit 340 then notifies the communication control unit 310 of a prompt that includes the generated sentences.

[0063] 9 is a second diagram showing a specific example of the process performed by the prompt generator 340. As described above, the prompt generator 340 Environmental data notified from the environmental data acquisition unit 320 (here, item values ​​of items that affect the air condition of the target environment), The environment language notified from the environment language generation unit 330; The system generates a prompt including a sentence expressing an image that co-occurs from the image, and notifies the communication control unit 310 of the prompt.

[0064] The example in Figure 9 is The environmental data acquisition unit 320 notifies the user of environmental data such as "location," "season," and "time," which affect the air quality of the target environment. The ambient language generation unit 330 notifies the user that "hot" is the ambient language. In this case, the prompt generation unit 340 generates a sentence that expresses an image that co-occurs from the environmental data and the environmental language. The prompt generation unit 340 notifies the communication control unit 310 of a prompt that includes the generated sentence.

[0065] 10 is a third diagram showing a specific example of the process performed by the prompt generator 340. As described above, the prompt generator 340 Environmental data notified from the environmental data acquisition unit 320 (here, item values ​​of items that affect the air condition of the target environment), The environment language notified from the environment language generation unit 330; The system generates a prompt including a sentence expressing an image that co-occurs from the image, and notifies the communication control unit 310 of the prompt.

[0066] The example in Figure 10 is The environmental data acquisition unit 320 notifies the user of the environmental data, such as the "location" and "time," which affect the air quality of the target environment. The ambient language generation unit 330 notifies the user that "it's very hot" is the ambient language. In this case, the prompt generation unit 340 generates a sentence that expresses an image that co-occurs from the environmental data and the environmental language. The prompt generation unit 340 notifies the communication control unit 310 of a prompt that includes the generated sentence.

[0067] 11 is a fourth diagram showing a specific example of the process performed by the prompt generator 340. As described above, the prompt generator 340 Environmental data notified from the environmental data acquisition unit 320 (here, item values ​​of items that affect the air condition of the target environment), The environment language notified from the environment language generation unit 330; A prompt including an image representing an image that co-occurs from the image is generated and notified to the communication control unit 310.

[0068] The example in Figure 11 is The environmental data acquisition unit 320 notifies the user of the environmental data, such as the "location" and "time," which affect the air quality of the target environment. The ambient language generation unit 330 notifies the user that "it's very hot" is the ambient language. 1 illustrates a situation in which prompt generation unit 340 generates an image that represents an image that co-occurs from environmental data and environmental language. Prompt generation unit 340 notifies communication control unit 310 of a prompt that includes the generated image.

[0069] 12 is a fifth diagram showing a specific example of processing by the prompt generation unit. As described above, prompt generation unit 340 adjusts image editing parameters based on the environmental data (here, item values ​​of items that affect the air condition of the target environment) notified from environmental data acquisition unit 320, includes the adjusted image editing parameters in a prompt, and notifies communication control unit 310.

[0070] The example of Figure 12 shows how prompt generation unit 340 adjusts image editing parameters based on environmental data when environmental data acquisition unit 320 notifies it of item values ​​of items that affect the air condition of the target environment, such as "season" or "time of day." As shown in the example of Figure 12, when environmental data acquisition unit 320 notifies it of item values ​​of items that affect the air condition of the target environment, such as "season" or "time of day," prompt generation unit 340 adjusts the image editing parameters "brightness," "contrast," "saturation," and "hue." In addition, prompt generation unit 340 adjusts the image editing parameters "filter," "blur," "sharpness," "shadow," "noise removal," and the like.

[0071] 13 is a sixth diagram showing a specific example of processing by the prompt generation unit. As described above, prompt generation unit 340 adjusts image editing parameters based on the environmental data (here, the item values ​​of the items indicating the air condition of the target environment) notified from environmental data acquisition unit 320, includes the adjusted image editing parameters in a prompt, and notifies communication control unit 310.

[0072] The example of Figure 13 shows how prompt generation unit 340 adjusts image editing parameters based on environmental data when environmental data acquisition unit 320 notifies it of item values ​​indicating the air condition of the target environment, such as "temperature" or "humidity." As shown in the example of Figure 13, when environmental data acquisition unit 320 notifies it of item values ​​indicating the air condition of the target environment, such as "temperature" or "humidity," prompt generation unit 340 adjusts the image editing parameters "brightness," "contrast," and "saturation." In addition, prompt generation unit 340 adjusts the image editing parameters "hue," "filter," "blur," "sharpness," "shadow" or "texture," "noise removal," and the like.

[0073] 14 is a seventh diagram showing a specific example of processing by the prompt generation unit. As described above, prompt generation unit 340 adjusts image editing parameters based on the environmental data (here, item values ​​of items indicating the air condition of the target environment) notified from environmental data acquisition unit 320, includes the adjusted image editing parameters in a prompt, and notifies communication control unit 310.

[0074] The example of Figure 14 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the user of the item values ​​of items indicating the air condition of the target environment, such as "wind speed" or "rainfall." As shown in the example of Figure 14, when the environmental data acquisition unit 320 notifies the user of the item values ​​of items indicating the air condition of the target environment, such as "wind speed" or "rainfall," the prompt generation unit 340 adjusts the image editing parameters "brightness," "contrast," and "saturation." The prompt generation unit 340 also adjusts the image editing parameters "hue," "filter," "blur," "sharpness," "effect" or "texture," "noise removal," and the like.

[0075] 15 is an eighth diagram showing a specific example of processing by the prompt generation unit. As described above, prompt generation unit 340 adjusts image editing parameters based on the environmental data (here, item values ​​of items that affect the air condition of the target environment) notified from environmental data acquisition unit 320, includes the adjusted image editing parameters in a prompt, and notifies communication control unit 310.

[0076] The example of Figure 15 shows how prompt generation unit 340 adjusts image editing parameters based on environmental data when environmental data acquisition unit 320 notifies it of item values ​​of items that affect the air condition of the target environment, such as "location," as environmental data. As shown in the example of Figure 15, when environmental data is notified it is item values ​​of items that affect the air condition of the target environment, such as "location," prompt generation unit 340 adjusts the image editing parameters "hue," "filter," and "texture." In addition, prompt generation unit 340 adjusts the image editing parameters "blur," "shadows and highlights," "noise," "brightness and contrast (or shadows)," and the like.

[0077] 16 is a ninth diagram showing a specific example of processing by the prompt generation unit. For example, when the prompt generation unit 340 generates a sentence to be included in a prompt, the prompt generation unit 340 further identifies an image style based on the generated sentence. Then, the prompt generation unit 340 adjusts image editing parameters based on the identified "image style."

[0078] The example in Figure 16 shows how the prompt generation unit 340 has identified the painting style as "watercolor style" and adjusted the image editing parameters "blur," "saturation," "add texture," "brush stroke," "hue," "transparency adjustment," etc.

[0079] The example in Figure 16 shows how the prompt generation unit 340 has identified the painting style as "oil painting style" and adjusted the image editing parameters "sharpness," "filter," "brush stroke," "color adjustment," "add texture," "shadows and highlights," etc.

[0080] The example in Figure 16 shows how the prompt generation unit 340 has identified the art style as "abstract painting" and adjusted the image editing parameters "saturation," "filter," "hue," "add noise," "blur and sharpness," "layer blend mode," etc.

[0081] Although the case where the prompt generation unit 340 identifies the "style" based on the sentence included in the prompt has been described above, the prompt generation unit 340 may also identify the "style" based on a user specification.

[0082] <Flow of image generation process by image generation system> A description will now be given of the flow of image generation processing by the image generation system 100. Fig. 17 is a first flowchart showing the flow of image generation processing by the image generation system.

[0083] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0084] In step S1702, the information processing device 120 generates an environmental language by converting the acquired environmental data into a natural language including sensory expressions.

[0085] In step S1703, the information processing device 120 generates a prompt based on either or both of the generated environment language and the acquired environment data. - Words or sentences expressing images that co-occur with either or both environmental data and environmental language; - Images that represent images that co-occur from either or both environmental data and environmental language; -Image editing parameters according to environmental data, etc. are included.

[0086] In step S1704, the information processing device 120 transmits the generated prompt to the server device 110. As a result, the generation AI 111 of the server device 110 generates an image according to the prompt.

[0087] In step S1705, the information processing device 120 acquires the image generated by the generation AI 111 of the server device 110 and displays it to the user.

[0088] In step S1706, the information processing device 120 determines whether or not to end the image generation process. If it is determined in step S1706 that the image generation process is to be continued (NO in step S1706), the process returns to step S1701.

[0089] On the other hand, if it is determined in step S1706 that the image generation process is to be ended (YES in step S1706), the image generation process is ended.

[0090] <Display example> A description will be given of a display example in which information processing device 120 displays an image generated by server device 110 based on a prompt generated by information processing device 120. Fig. 18 is a first diagram showing an example of a display screen.

[0091] 18, display screen 1800 has area 1810 for displaying an image, area 1820 for displaying environmental data, area 1830 for displaying image editing parameters, and area 1840 for displaying text included in the prompt used to generate the image. Note that display screen 1800 may have an area for displaying environmental language instead of or in addition to area 1820 for displaying environmental data.

[0092] <Summary> As is clear from the above description, the information processing device 120 according to the first embodiment: Obtain environmental data including item values ​​of items that indicate the air quality of the target environment or items that affect the air quality of the target environment. -Generate environmental language by converting environmental data into natural language that includes sensory expressions. ·Generating prompts that include phrases or sentences or images that express images that co-occur from environmental language or environmental data. Use the prompt to operate the generation AI and display an image generated by the generation AI showing the air quality of the target environment.

[0093] As a result, the information processing device 120 according to the first embodiment can generate an image that intuitively expresses the state of the air in the target environment.

[0094] [Second embodiment] In the first embodiment, an image intuitively representing the air condition of the target environment is displayed on the display screen 1800. However, the image displayed on the display screen is not limited to an image intuitively representing the air condition of the target environment. For example, an image intuitively representing the opposite air condition (air condition of a comparison environment) to the air condition of the target environment may be displayed for comparison. This allows the user to more clearly grasp the air condition of the target environment. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0095] <Functional configuration of information processing device> The functional configuration of an information processing device 120 according to the second embodiment will be described. Fig. 19 is a second diagram showing an example of the functional configuration of an information processing device. The difference from the functional configuration described with reference to Fig. 3 in the first embodiment is that comparative environment data is input to an environment data acquisition unit 320 from a comparative environment data storage unit 1910.

[0096] The comparative environment data storage unit 1910 stores the comparative environment data. The environmental data acquisition unit 320 acquires each item included in the environmental data notified from the communication control unit 310, and Each item included in the environmental data measured by the sensors 1 to n (reference numerals 130_1 to 130_n), and Each item included in the environmental data entered by the user, It has similar items.

[0097] However, the item values ​​of each item in the comparative environmental data are as follows: The item values ​​of each item included in the environmental data notified from the communication control unit 310, and Item values ​​of each item included in the environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), and -Item values ​​of each item included in the environmental data entered by the user; The comparison environment data has different item values ​​(for example, item values ​​in a poor environment). Specifically, the item values ​​of each item in the comparison environment data are item values ​​that indicate an air condition opposite to the air condition in the target environment (the air condition in the comparison environment), or item values ​​that affect the air condition opposite to the air condition in the target environment (the air condition in the comparison environment).

[0098] The environmental data acquisition unit 320 acquires environmental data, notifies the acquired environmental data to the environmental language generation unit 330 and the prompt generation unit 340, and reads out comparative environmental data. The environmental data acquisition unit 320 notifies the environmental language generation unit 330 and the prompt generation unit 340 of the read out comparative environmental data.

[0099] The prompt generator 340 generates a prompt for generating an image that sensorily expresses the air condition of the comparison environment, in addition to a prompt for generating an image that sensorily expresses the air condition of the target environment. This enables the output unit 350 to obtain the image that sensorily expresses the air condition of the target environment and the image that sensorily expresses the air condition of the comparison environment from the server device 110 via the communication controller 310 and display them in comparison on the display screen.

[0100] <Display example> A description will be given of a display example in which information processing device 120 displays an image generated by server device 110 based on a prompt generated by information processing device 120. Fig. 20 is a second diagram showing an example of a display screen.

[0101] 20, display screen 2000 has area 2010 for displaying an image that intuitively expresses the air condition of the target environment, and area 2011 for displaying a sentence included in the prompt used to generate the image. Display screen 2000 also has area 2020 for displaying an image that intuitively expresses the air condition of a comparison environment, and area 2021 for displaying a sentence included in the prompt used to generate the image.

[0102] In this way, by displaying an image that intuitively expresses the air condition of the target environment and an image that intuitively expresses the air condition of the comparison environment, for example, the user can more clearly understand that the air condition of the target environment is the air condition of an ideal environment.

[0103] In the example of Fig. 20, an image that intuitively represents the air condition of a poor environment is displayed as an image that intuitively represents the air condition of a comparison environment. However, if the target environment is a poor environment, an image that intuitively represents the air condition of an ideal environment may be displayed. This makes it possible to provide a user interface that, for example, when a user selects an image that intuitively represents the air condition of the ideal environment, controls the air conditioning to create the ideal environment.

[0104] <Summary> As is clear from the above description, the information processing device 120 according to the second embodiment: When displaying an image showing the air quality of a target environment, an image showing the air quality of a comparison environment different from the target environment is also displayed.

[0105] As a result, according to the second embodiment, the user can more clearly grasp the state of the air in the target environment.

[0106] [Third embodiment] In the first and second embodiments, the information processing device 120 causes the generation AI 111 to generate a still image. However, the information processing device 120 may cause the generation AI 111 to generate a moving image. The following describes the third embodiment, focusing on differences from the first embodiment.

[0107] <Functional configuration of information processing device> The functional configuration of an information processing device 120 according to the third embodiment will be described. Fig. 21 is a third diagram showing an example of the functional configuration of an information processing device. The functional configuration shown in Fig. 21 differs from the functional configuration described with reference to Fig. 3 in the first embodiment in that it includes a behavior control language generation unit 2110.

[0108] The control table storage unit 2120 stores a control table. The control table mainly has items similar to the items included in the environmental data notified to the environmental data acquisition unit 320 by the communication control unit 310. The control table also has an action control language that indicates the level of action of each component in the video, associated with the item value of each item. The component here refers to the component (e.g., clouds, plants, flowers, rivers, etc.) included in the video generated by the generation AI.

[0109] The movement control language generation unit 2110 references the control table stored in the control table storage unit 2120 to generate a movement control language based on the environmental data notified by the environmental data acquisition unit 320. The movement control language generation unit 2110 notifies the prompt generation unit 340 of the generated movement control language.

[0110] The prompt generation unit 340 adds the action control language notified by the action control language generation unit 2110 to the wording or sentence included in the prompt.

[0111] <Specific example of processing by the motion control language generation unit> Of the units of the information processing device 120 according to the third embodiment, a specific example of processing by the movement control language generation unit 2110 will be described. Fig. 22 is a diagram showing a specific example of processing by the movement control language generation unit. As shown in Fig. 22, the control table storage unit 2120 stores, for example, control tables 2210 to 2230.

[0112] Control table 2210 is an example of a table that defines the action control language when the component is "cloud." The example of control table 2210 shows how the action control language is defined for each item value of the item "wind speed" and for each altitude.

[0113] Control table 2220 is an example of a table that defines the action control language when the component is "plants, trees, flowers." The example of control table 2220 shows how the action control language is defined for each item value of the item="wind speed on the ground."

[0114] Control table 2230 is an example of a table that defines the action control language when the component is a "river." The example of control table 2230 shows how the action control language is defined for each item value of the item="rainfall."

[0115] For example, when the environmental data acquisition unit 320 notifies the movement control language generation unit 2110 of the wind speed at a specific altitude as environmental data, the movement control language generation unit 2110 refers to the control table 2210. The movement control language generation unit 2110 generates movement control language that indicates the degree of movement (flow) of clouds according to the wind speed at the specific altitude, and notifies the prompt generation unit 340. This enables the prompt generation unit 340 to add the movement control language to the sentence component "cloud" included in the prompt.

[0116] For example, when the environmental data acquisition unit 320 notifies the movement control language generation unit 2110 of the wind speed on the ground as environmental data, the movement control language generation unit 2110 refers to the control table 2220. The movement control language generation unit 2110 generates a movement control language that indicates the degree of movement (swaying) of plants and flowers according to the wind speed on the ground, and notifies the prompt generation unit 340. This allows the prompt generation unit 340 to add the movement control language to the sentence component "plants, flowers" included in the prompt.

[0117] For example, when the environmental data acquisition unit 320 notifies the movement control language generation unit 2110 of the amount of rainfall as environmental data, the movement control language generation unit 2110 refers to the control table 2220. The movement control language generation unit 2110 generates movement control language that indicates the degree of movement (flow) of the river depending on the amount of rainfall, and notifies the prompt generation unit 340. This allows the prompt generation unit 340 to add movement control language when the sentence component included in the prompt contains the character "river".

[0118] <Examples of prompts and video images> An example of a prompt generated by the information processing device 120 according to the third embodiment and an example of a moving image generated by the generation AI 111 of the server device 110 using the prompt will be described below. Fig. 23 is a diagram showing an example of a prompt (sentence) and an example of a moving image.

[0119] Of these, reference numeral 2310 indicates that the generated prompt includes "clouds" as a component and "quickly" is added as an action control word. Reference numeral 2311 indicates an example of a moving image (a moving image showing clouds passing by quickly) generated by the generation AI 111 of the server device 110 based on a prompt including the sentence indicated by reference numeral 2310.

[0120] Reference numeral 2320 indicates that the generated prompt includes the components "river," "flower," and "cloud," but no motion control language is added to any of the components. In this case, generation AI 111 of server device 110 generates a video in which each component is operated based on the default motion control language. Reference numeral 2321 indicates an example of a video (a video showing a river and clouds flowing at their default speeds, and flowers swaying at their default speeds) generated by server device 110 based on a prompt including the sentence indicated by reference numeral 2320.

[0121] Reference numeral 2330 indicates that the generated prompt includes "river" as a component and "strongly" is added as an action control word. Reference numeral 2331 indicates an example of a video (a video showing a vigorously flowing river) generated by the generation AI 111 of the server device 110 based on a prompt including the sentence shown by reference numeral 2330.

[0122] <Flow of image generation process by image generation system> The flow of image generation processing by the image generation system 100 including the information processing device 120 according to the third embodiment will be described below. Fig. 24 is a second flowchart showing the flow of image generation processing by the image generation system.

[0123] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0124] In step S1702, the information processing device 120 generates an environmental language by converting the acquired environmental data into a natural language including sensory expressions.

[0125] In step S2401, the information processing device 120 generates a motion control language based on the acquired environmental data.

[0126] In step S1703, the information processing device 120 generates a prompt based on either or both of the generated environment language and the acquired environment data. The information processing device 120 adds corresponding action control language to the components of the sentence included in the generated prompt. Note that the prompt generated by the information processing device 120 may include: - Words or sentences expressing images that co-occur from either or both of environmental data and environmental language (components are accompanied by action control language); - Images that represent images that co-occur from either or both environmental data and environmental language; -Image editing parameters according to environmental data, etc. are included.

[0127] In step S2403, the information processing device 120 transmits the generated prompt to the server device 110. As a result, the generation AI 111 of the server device 110 generates an image (moving image) according to the prompt.

[0128] In step S2404, the information processing device 120 acquires the image (moving image) generated in the generation AI 111 of the server device 110, and displays it to the user.

[0129] In step S1706, the information processing device 120 determines whether or not to end the image generation process. If it is determined in step S1706 that the image generation process is to be continued (NO in step S1706), the process returns to step S1701.

[0130] On the other hand, if it is determined in step S1706 that the image generation process is to be ended (YES in step S1706), the image generation process is ended.

[0131] <Summary> As is clear from the above description, the information processing device 120 according to the third embodiment performs the following processes in addition to the processes performed by the information processing device 120 according to the first embodiment: Generate a behavior control language that expresses the degree of behavior of the component based on the environmental data. · Add action control language to sentence components included in the prompt. - Operate the generation AI using the prompt, and display a moving image showing the air condition of the target environment generated by the generation AI, in which the components operate based on the operation control language.

[0132] As a result, the information processing device 120 according to the third embodiment can generate a moving image that intuitively expresses the state of the air in the target environment.

[0133] [Fourth embodiment] In each of the above embodiments, an image is acquired by operating the generation AI 111 included in the server device 110. In contrast, in the present embodiment and subsequent embodiments, an image is acquired by operating the generation AI 111 included in the server device 110 in a generation phase after additional learning has been performed in a learning phase. The following describes the fourth embodiment, focusing on the differences from the above embodiments.

[0134] <System configuration of image generation system> The system configuration in the learning phase of an image generation system to which an information processing device according to the fourth embodiment is applied will be described. Fig. 25 is a second diagram showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the learning phase.

[0135] 25, an image generation system 2500 includes a server device 110, an information processing device 120, and a server device 2540. In the image generation system 2500, the information processing device 120, the server device 110, and the server device 2540 are communicably connected via a network 150.

[0136] In the learning phase, the server device 110 has a generation AI 111 and a fine-tuning unit 2511. When the server device 110 receives learning data from the information processing device 120 via the network 150, the server device 110 uses the fine-tuning unit 2511 to perform additional learning on the generation AI 111. As a result, the fine-tuning unit 2511 generates tuned models corresponding to each piece of learning data.

[0137] In the learning phase, the information processing device 120 generates learning data and instructs the generation AI 111 to perform additional learning. Specifically, the information processing device 120 acquires a group of images (an example of a group of specific images) to be included in the learning data from the server device 2540. The information processing device 120 generates learning data including the acquired group of images and captions indicating each component included in each image. The information processing device 120 transmits the generated learning data to the server device 110. The information processing device 120 instructs the generation AI 111 to perform additional learning using a model (LoRA in this embodiment) for generating each image (an example of each specific image) included in the generated learning data.

[0138] In the learning phase, the information processing device 120 calculates a component-specific score for each piece of learning data based on the score indicating the importance of each component in the image, and stores the score in association with the learning data.

[0139] The server device 2540 functions as an information providing unit 2541 and provides images to be included in the learning data to the information processing device 120 via the network 150.

[0140] <Functional configuration of information processing device> The functional configuration of the information processing device 120 in the learning phase will be described. Fig. 26 is a fourth diagram showing an example of the functional configuration of the information processing device. An information processing program is installed in the information processing device 120, and by executing the information processing program in the learning phase, the information processing device 120: A communication control unit 310, Image data collection unit 2610; a caption generator 2620; A learning data generation unit 2630, Additional learning section 2640, It functions as:

[0141] The communication control unit 310 acquires images from the server device 2540 via the network 150 and notifies the image data collection unit 2610. The communication control unit 310 acquires learning data and instructions for additional learning from the additional learning unit 2640 and transmits them to the server device 110 via the network 150.

[0142] The image data collection unit 2610 stores the image notified by the communication control unit 310 in the image data storage unit 2650 .

[0143] The caption generation unit 2620 reads images stored in the image data storage unit 2650 and performs image recognition processing on the read images to recognize each component included in the read images and calculate the reliability of each recognized component. The caption generation unit 2620 then generates captions indicating each component included in each image. The caption generation unit 2620 notifies the learning data generation unit 2630 of the read images, the generated captions, and the calculated reliability of each component.

[0144] The learning data generation unit 2630 acquires the images, captions, and reliability of each component element notified by the caption generation unit 2620. The learning data generation unit 2630 classifies the images notified by the caption generation unit 2620 into multiple groups according to the type of image content. The learning data generation unit 2630 calculates a score for each component element for each group based on the reliability of each component element corresponding to each image belonging to the classified group.

[0145] The training data generation unit 2630 generates training data including images belonging to the classified groups and captions corresponding to the images. The training data generation unit 2630 associates the generated training data with the scores for each component element calculated for the group, and stores the data in the training data storage unit 2660.

[0146] The additional learning unit 2640 sequentially reads out the learning data generated for each group and notifies the communication control unit 310 of the data together with an instruction for additional learning. This allows the communication control unit 310 to sequentially transmit the read learning data to the server device 110 and instruct the server device 110 to perform additional learning on the generated AI 111 using the generated learning data.

[0147] <Specific examples of processing by each unit of the information processing device> A specific example of processing by each unit (here, the caption generation unit 2620 and the learning data generation unit 2630) of the information processing device 120 will be described.

[0148] (1) Specific examples of processing by the caption generation unit 27 is a diagram showing a specific example of processing by the caption generation unit. As described above, the caption generation unit 2620 reads an image stored in the image data storage unit 2650 and performs image recognition processing on the read image to recognize each component included in the read image and calculate the reliability of each recognized component.

[0149] 27, reference numeral 2710 denotes an image read by the caption generation unit 2620 from the image data storage unit 2650. In FIG. 27, reference numeral 2720 denotes an image generated by performing image recognition processing on the image read by the caption generation unit 2620. "sunsetcloud", "scenery", "sun", "sky", "cloud", "mountain", 27 shows how each of the components such as the above has been recognized. Also, in Fig. 27, reference numeral 2720 shows how the caption generation unit 2620 calculates the reliability of each component when performing image recognition processing on the read image. The caption generation unit 2620 notifies the learning data generation unit 2630 of the read image, captions indicating each component included in the image, and the calculated reliability of each component.

[0150] (2) Specific examples of processing by the learning data generation unit 28 is a diagram showing a specific example of processing by the learning data generation unit. As described above, the learning data generation unit 2630 acquires images, captions indicating each component included in the image, and the reliability of each component from the caption generation unit 2620, and classifies the acquired images into multiple groups according to the type of image content. The learning data generation unit 2630 calculates a score for each component for each group based on the reliability of each component corresponding to each image belonging to the classified group.

[0151] 28, reference numeral 2710 denotes an image notified by the caption generation unit 2620, and reference numeral 2720 denotes a caption notified by the caption generation unit 2620. Reference numeral 2730 denotes the reliability of each component element notified by the caption generation unit 2620.

[0152] In FIG. 28, reference numeral 2800 denotes the learning data generation unit 2630. From the images shown in the reference numeral 2710, images of the type of image content "sunset" are extracted and classified into a group of the type of image content "sunset" (see the reference numeral 2810), Associate a caption corresponding to an image of image content type="sunset" with an image of image content type="sunset" (see reference numeral 2820); This shows how training data (for sunsets) was generated.

[0153] In FIG. 28, reference numeral 2830 denotes the learning data generation unit 2630. From the reliability of each component element shown by the reference numeral 2730, the reliability of each component element corresponding to the image belonging to the group of image content type="sunset" is read out, - Add up the reliability of each component and divide by the number of images in the group with image content type = "sunset" In this example, the component scores are calculated and associated with the learning data (for sunsets). Note that when associating the component scores with the learning data (for sunsets), components with scores below a predetermined threshold may be excluded.

[0154] <Server equipment details> The following describes details of the generation AI 111 and fine tuning unit 2511 of the server device 110 in the fourth embodiment. Fig. 29 is a diagram showing details of the generation AI and fine tuning unit.

[0155] 29, the generation AI 111 includes a VAE (Variational Auto-Encoder) 2911, an encoder 2912, a decoder 2913, and a VAE 2914. The fine tuning unit 2511 includes a LoRA (Low-Rank Adaptation) 2920. The LoRA 2920, for example, LoRA2921 for sunsets, which is used for additional learning of AI111 generated using training data (for sunsets), LoRA2922 for sardine clouds, which is used for additional learning of AI111 generated using training data (for sardine clouds), LoRA2923 for landscapes, which is used for additional learning of the generation AI using learning data (for landscapes), It has.

[0156] For example, if you use LoRA2921 for sunset to perform additional learning on the generation AI, The image indicated by reference numeral 2810 of the learning data (for sunset) indicated by reference numeral 2800 in FIG. The model parameters of the generation AI 111 are fine-tuned so that the image reproduced by the generation AI 111 matches the image shown by reference numeral 2810 (the image input to the generation AI 111).

[0157] <Flow of additional learning process by image generation system> A description will be given of the flow of additional learning processing in the learning phase by the image generation system 2500. Fig. 30 is a flowchart showing the flow of additional learning processing by the image generation system.

[0158] In step S3001, the information processing device 120 collects images provided by the server device 2540.

[0159] In step S3002, the information processing device 120 performs image recognition processing on the collected image to recognize each component included in the image and calculate the reliability of each recognized component. The information processing device 120 generates a caption indicating each component included in the image.

[0160] In step S3003, the information processing device 120 generates groups of additional learning units by classifying the collected images into a plurality of groups according to the type of image content.

[0161] In step S3004, the information processing device 120 calculates a component-specific score for each group based on the reliability of each component.

[0162] In step S3005, the information processing device 120 generates learning data in which images and captions are associated with each other for each group of additional learning units.

[0163] In step S3006, the information processing device 120 sequentially transmits the learning data generated for each group of additional learning units to the server device 110, thereby performing additional learning on the generated AI 111 using each LoRA possessed by the fine tuning unit 2511.

[0164] In step S3007, the information processing device 120 determines whether or not to end the additional learning process. If it is determined in step S3007 that the additional learning process is to be continued (NO in step S3007), the process returns to step S3001.

[0165] On the other hand, if it is determined in step S3007 that the additional learning process is to be ended (YES in step S3007), the additional learning process is ended.

[0166] <System configuration of image generation system> A system configuration in the generation phase of an image generation system to which an information processing device according to the fourth embodiment is applied will be described. Fig. 31 is a third diagram showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the generation phase.

[0167] 31, an image generation system 3100 includes a server device 110, an information processing device 120, and a server device 140. In the image generation system 3100, the information processing device 120, the server device 110, and the server device 140 are communicably connected via a network 150.

[0168] In the generation phase, the server device 110 has a generation AI 111 and a tuned model 3110. In the generation phase, when the server device 110 receives a prompt from the information processing device 120 via the network 150, the server device 110 operates the generation AI 111 using the tuned model 3110 to generate an image according to the prompt. The server device 110 transmits the generated image to the information processing device 120 via the network 150.

[0169] The information processing device 120 is the same as the information processing device 120 described in the first embodiment with reference to FIG. 1, and therefore a description thereof will be omitted here. However, in the case of the information processing device 120 according to the fourth embodiment, the prompt sent to the server device 110 includes the following: Specify the tuned model 3110 (execution LoRA) to use, -Weights when using the tuned model (weights according to the type of image content for each of the multiple learning data (weights for each execution LoRA)), shall be included.

[0170] The server device 140 is the same as the server device 140 described in the first embodiment with reference to FIG. 1, and therefore a description thereof will be omitted here.

[0171] <Functional configuration of information processing device> The functional configuration of the information processing device 120 according to the fourth embodiment in the generation phase will be described. Fig. 32 is a fifth diagram showing an example of the functional configuration of the information processing device. The difference from the functional configuration described with reference to Fig. 3 in the first embodiment is that a weight calculation unit 3210 is included.

[0172] The weight calculation unit 3210 obtains the wording or sentences included in the prompt generated by the prompt generation unit 340, and extracts the components included in each training data. The weight calculation unit 3210 reads the component-specific scores associated with the training data stored in the training data storage unit 2660, and extracts the components corresponding to the read component-specific scores from the prompt.

[0173] The weight calculation unit 3210 calculates the sum (total score) of the scores of each component element for each learning data item from the scores of each learning data item corresponding to the component elements extracted from the prompt.

[0174] The weight calculation unit 3210 compares the total scores of each training data and selects the training data with the highest total score to designate the corresponding LoRA as the active LoRA. The weight calculation unit 3210 calculates a weight for each active LoRA based on the total score of each selected training data.

[0175] The weight calculation unit 3210 notifies the prompt generation unit 340 of the weight of each active LoRA. This allows the prompt generation unit 340 to generate a prompt that includes the designation of the active LoRA and the weight of each active LoRA.

[0176] As a result, the information processing device 120 according to the fourth embodiment can select a generated AI that has undergone additional training in advance using each of a plurality of pieces of training data based on the generated prompt, and operate it based on the assigned weights. Note that "selecting a generated AI that has undergone additional training in advance" here refers to, for example, selecting a combination of LoRAs to be executed in a generated AI that includes a plurality of LoRAs that have been generated by additional training in advance.

[0177] <Specific example of processing by weight calculation unit> A specific example of the processing by the weight calculation unit 3210 will be described below. Fig. 33 is a first diagram showing a specific example of the processing by the weight calculation unit.

[0178] In FIG. 33, reference numeral 3310 denotes an example of a sentence included in a prompt generated by the prompt generation unit 340.

[0179] The weight calculation unit 3210 acquires the sentence (reference numeral 3310) included in the prompt, and extracts from the acquired sentence the components corresponding to the component-specific scores (reference numerals 3331 to 3333) associated with each learning data. In the sentence (reference numeral 3310) included in the prompt, the underlined components indicate the components extracted by the weight calculation unit 3210 as the components corresponding to the component-specific scores (reference numerals 3331 to 3333) associated with each learning data.

[0180] The weight calculation unit 3210 calculates the sum (total score) of the scores of each component element for each learning data from the scores of each learning data corresponding to the component elements (underlined in reference numeral 3310) extracted from the prompt. - The total score = "399" was calculated from the score corresponding to the component extracted from the prompt (underlined code 3310) among the component scores (code 3331) associated with the learning data (for sardine clouds). The total score = "376" was calculated from the score corresponding to the component extracted from the prompt (underlined code 3310) among the component scores (code 3332) associated with the learning data (for sunsets). The total score = "347" was calculated from the score corresponding to the component extracted from the prompt (underlined code 3310) among the component scores (code 3333) associated with the learning data (for landscape paintings). This shows:

[0181] The weight calculation unit 3210 compares the total scores of each piece of training data and selects the training data with the highest total score, thereby designating the corresponding LoRA as the active LoRA. In the example of Fig. 33, due to space limitations, the total scores of three pieces of training data are calculated, and the three pieces of training data are selected, thereby designating the LoRA for mackerel clouds, the LoRA for sunsets, and the LoRA for landscape painting as the active LoRAs (see reference numeral 3322).

[0182] The weight calculation unit 3210 calculates a weight for each LoRA based on the total score of each selected training data. In Fig. 33, reference numeral 3323 indicates the weight of each execution LoRA corresponding to each training data based on the ratio of the total scores of the training data (sardine cloud), training data (sunset), and training data (landscape). 0.34, 0.36, 0.30, The figure shows how the calculation was done.

[0183] The weight calculation unit 3210 notifies the prompt generation unit 340 of the weight of each active LoRA. ·LoRA weight for sardine cloud = 0.34, ·LoRA weight for sunset=0.36, ·LoRA weight for landscape=0.30, 10 shows how the prompt generator 340 is notified of the above.

[0184] <Flow of image generation process by image generation system> The flow of image generation processing in the generation phase by the image generation system 3100 will be described. Fig. 34 is a third flowchart showing the flow of image generation processing by the image generation system. Note that the difference from the first flowchart shown in Fig. 17 is step S3401.

[0185] In step S3401, the information processing device 120 extracts components corresponding to the component-specific scores associated with each training data from the sentence included in the generated prompt. The information processing device 120 calculates the sum (total score) of the scores for each component for each training data from the scores corresponding to the extracted components. The information processing device 120 designates an execution LoRA based on the total score for each training data, and calculates a weight for each execution LoRA. The information processing device 120 includes the designation of the execution LoRA and the weight for each execution LoRA in the prompt.

[0186] <Summary> As is clear from the above description, the information processing device 120 according to the fourth embodiment performs the following in the learning phase: - Recognize each component contained in the collected images and calculate the reliability of each recognized component. The collected images are classified into multiple groups according to the type of image content, and component scores are calculated for the components contained in the images belonging to each group. For each group, training data is generated in which images are associated with captions indicating each component element contained in the image, and a component-specific score is associated with each training data. Using the multiple generated training data, the corresponding LoRA is used to perform additional training on the generating AI to generate a tuned model.

[0187] Furthermore, the information processing apparatus 120 according to the fourth embodiment performs the following in the generation phase: - Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. ·Generating prompts that include phrases or sentences or images that express images that co-occur from environmental language or environmental data. Based on the sentence components contained in the generated prompt, the execution LoRA is specified and the weight of each execution LoRA is calculated. By using a prompt that specifies the LoRAs to be executed and includes the weight of each LoRA, the generation AI is operated using the execution LoRAs. Also, by operating the generation AI, an image showing the air quality of the target environment is generated by the generation AI and displayed to the user.

[0188] In this way, by operating a generation AI that has undergone additional learning using multiple learning data generated for each type of image content, the information processing device 120 according to the fourth embodiment can generate images that intuitively express the state of the air with high accuracy.

[0189] [Fifth embodiment] In the fourth embodiment, the weight of the execution LoRA is calculated by referring to the component-specific scores associated with the training data. However, the calculation method of the weight of the execution LoRA is not limited to this, and the weight of the execution LoRA may be calculated by other calculation methods. The following describes the fifth embodiment, focusing on the differences from the fourth embodiment.

[0190] <Specific example of processing by weight calculation unit> A specific example of processing by the weight calculation unit 3210 of the information processing device 120 according to the fifth embodiment will be described. Fig. 35 is a second diagram showing a specific example of processing by the weight calculation unit. The differences from the first diagram shown in Fig. 33 are reference numerals 3421 and 3422.

[0191] As indicated by the reference numeral 3421, in the information processing device 120 according to the fifth embodiment, the weight calculation unit 3210 uses LLM to determine the importance of the word represented by the execution LoRA in the sentence included in the prompt. Note that LLM is an abbreviation for Large Language Model, and refers to a large-scale language model.

[0192] The example in Figure 35 shows a case where the LoRA for sardine clouds, the LoRA for sunset, and the LoRA for landscape are specified as the execution LoRAs. The example in Figure 35 also shows a case where the importance of the words represented by the specified execution LoRAs, "sardine clouds," "sunset," and "landscape," in the sentence (reference numeral 3310) included in the prompt is determined.

[0193] According to the LLM, the "sunset" is "an element that determines the overall atmosphere, and influences the other elements with light and color," so its importance is determined to be "50%." According to the LLM, the "sardine clouds" are "beautifully illuminated by the light of the sunset, and their shadows add depth to the landscape," so their importance is determined to be "30%." According to the LLM, the "landscape painting" is "further enhanced by the sunset and sardine clouds, creating an overall beauty, but does not play as central a role as the sunset and sardine clouds," so its importance is determined to be "20%."

[0194] As a result, the weight calculation unit 3210 calculates the weights of the respective execution LoRAs as 0.5, 0.3, and 0.2.

[0195] In the information processing device 120 according to the fifth embodiment, the weight calculation unit 3210 notifies the prompt generation unit 340 of the weight of each execution LoRA. ·LoRA weight for sardine cloud = 0.5, ·LoRA weight for sunset=0.3, ·Landscape LoRA weight=0.2, 10 shows how the prompt generator 340 is notified of the above.

[0196] <Summary> As is clear from the above description, the information processing device 120 according to the fifth embodiment performs the following in the generation phase: - Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. ·Generating prompts that include phrases or sentences or images that express images that co-occur from environmental language or environmental data. · Specify the LoRA to be performed based on the sentence components contained in the generated prompt. The weight of each execution LoRA is calculated by using LLM to determine the importance of the words represented by each execution LoRA in the sentence included in the prompt. By running the generation AI using a prompt that includes the specification of the LoRA to be executed and the weight of each LoRA to be executed, an image showing the air quality of the target environment generated by the generation AI is displayed.

[0197] In this way, by operating a generation AI that has undergone additional learning using learning data generated for each type of image content, the information processing device 120 of the fifth embodiment can generate images that intuitively express the state of air with high accuracy, as in the fourth embodiment.

[0198] [Sixth embodiment] In the fourth and fifth embodiments, the LoRA to be executed is specified based on the sentence included in the generated prompt. However, the method of specifying the LoRA to be executed is not limited to this. For example, conditions for specifying the LoRA to be executed (referred to as environmental conditions) may be set, and the LoRA to be executed may be specified when the environmental conditions are met. The sixth embodiment will be described below, focusing on the differences from the fourth and fifth embodiments.

[0199] <Functional configuration of information processing device> The functional configuration of the information processing device 120 according to the sixth embodiment in the generation phase will be described. Fig. 36 is a sixth diagram showing an example of the functional configuration of the information processing device. The functional configuration differs from that described with reference to Fig. 32 in the fourth embodiment in that a selection unit 3610 is provided instead of the weight calculation unit 3210.

[0200] The selection unit 3610 acquires the wording or sentences included in the prompt generated by the prompt generation unit 340, and extracts the components included in each training data. The selection unit 3610 reads the component-specific scores associated with the training data stored in the training data storage unit 2660, and extracts the components corresponding to the read component-specific scores from the prompt.

[0201] The selection unit 3610 calculates the sum (total score) of the scores of each component element for each learning data item from the scores of each learning data item corresponding to the component elements extracted from the prompt.

[0202] The selection unit 3610 compares the total scores of each training data and selects the training data with the highest total score to designate the corresponding LoRA as the execution LoRA. At this time, the selection unit 3610 determines whether or not environmental conditions are assigned to the selected training data, and if environmental conditions are assigned, determines whether or not the sentences included in the generated prompt are consistent with the environmental conditions.

[0203] If the selection unit 3610 determines that the selected learning data has environmental conditions attached to it and that the sentences contained in the generated prompt are consistent with the environmental conditions, it does not change the specified execution LoRA.

[0204] On the other hand, if it is determined that the selected learning data has environmental conditions attached to it and that the sentence contained in the generated prompt is not consistent with the environmental conditions, the selection unit 3610 re-designates the LoRA corresponding to other learning data as the execution LoRA.

[0205] Alternatively, if it is determined that the selected learning data does not have an environmental condition, the selection unit 3610 re-designates a LoRA corresponding to any other learning data as the execution LoRA.

[0206] The selector 3610 distributes weights equally among the designated or redesignated LoRAs to be executed, and notifies the prompt generator 340 of the weights distributed equally among the designated LoRAs to be executed.

[0207] <Specific example of processing by the selection unit> A specific example of the processing performed by the selection unit 3610 will be described below. Fig. 37 is a diagram showing a specific example of the processing performed by the selection unit.

[0208] In FIG. 37, reference numeral 3310 is the same as reference numeral 3310 shown in FIG. 33, and reference numerals 3321 and 3322 are the same as reference numerals 3321 and 3322 shown in FIG. 33, and therefore description thereof will be omitted here.

[0209] As indicated by reference numeral 3721, the selection unit 3610 determines whether or not environmental conditions are assigned to the selected learning data. Here, it is assumed that the selection unit 3610 has selected learning data (for sardine clouds), learning data (for sunsets), and learning data (for landscapes). Furthermore, as indicated by reference numerals 3731 to 3733, The learning data (for sardine clouds) does not include environmental conditions. The learning data (for sunsets) includes the environmental conditions of time = evening and weather = sunny. The learning data (for landscape images) includes the location "anywhere" as an environmental condition. Let's say.

[0210] In this case, the selection unit 3610 The training data (for sunsets) has environmental conditions attached to it, and if the sentences included in the prompt are determined to satisfy the attached environmental conditions, the corresponding sunset LoRA will not be changed as the execution LoRA. The learning data (for landscapes) has environmental conditions attached to it, and if the sentences included in the prompt are determined to satisfy the attached environmental conditions, the corresponding landscape LoRA will not be changed as the execution LoRA. · Since the learning data (for sardine clouds) does not have environmental conditions assigned, any other LoRA, the LoRA for temperature, is re-designated as the execution LoRA (see reference number 3721).

[0211] As indicated by reference numeral 3722, the selection unit 3610 equally distributes weights (for example, 0.3 each) to each execution LoRA (here, the LoRA for sunset, the LoRA for landscape, and the LoRA for temperature).

[0212] As indicated by the reference numeral 3723, the selection unit 3610 selects the weight (an example of a predetermined weight) of each designated execution LoRA or each re-designated execution LoRA as follows: ·LoRA weight for sunset=0.3, ·LoRA weight for landscape=0.3, ·LoRA weight for temperature = 0.3, is notified to the prompt generator 340.

[0213] <Summary> As is clear from the above description, the information processing device 120 according to the sixth embodiment performs the following in the generation phase: - Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. ·Generating prompts that include phrases or sentences or images that express images that co-occur from environmental language or environmental data. · Specify the LoRA to be performed based on the sentence components contained in the generated prompt. - Determine whether the sentences contained in the generated prompt satisfy the environmental conditions assigned to the training data, and if they do not, re-specify the LoRA to be executed. · Weights are evenly distributed for each LoRA run. By using a prompt that specifies the LoRAs to be executed and includes the weight of each LoRA, the generation AI is operated using the execution LoRAs. Also, by operating the generation AI, an image showing the air quality of the target environment generated by the generation AI is obtained and displayed to the user.

[0214] In this way, the information processing device 120 according to the sixth embodiment operates a generation AI that has undergone additional learning using learning data generated for each type of image content, taking into account the environmental conditions assigned to the learning data and based on predetermined weights. As a result, the information processing device 120 according to the sixth embodiment can appropriately generate an image that intuitively expresses the state of the air.

[0215] [Seventh embodiment] In the fourth embodiment, the weight calculation unit 3210 calculates the weight of each execution LoRA based on the total score. In contrast, in the seventh embodiment, multiple combinations of weights for each execution LoRA are prepared, and the generation AI is operated using each execution LoRA under each combination, and the user evaluates each generated image. This makes it possible to generate images using an appropriate combination of weights. The seventh embodiment will be described below, focusing on the differences from the fourth embodiment.

[0216] <Functional configuration of information processing device> The functional configuration of the information processing device 120 according to the seventh embodiment in the generation phase will be described. Fig. 38 is a seventh diagram showing an example of the functional configuration of the information processing device. The differences from the functional configuration described using Fig. 32 in the above fourth embodiment are that the function of the weight calculation unit 3810 is different from the function of the weight calculation unit 3210 in Fig. 32, and that the information processing device 120 according to the seventh embodiment has a weight storage unit 3820.

[0217] The weight calculation unit 3810 obtains the wording or sentences included in the prompt generated by the prompt generation unit 340, and extracts the components included in each training data. The weight calculation unit 3810 reads the component-specific scores associated with the training data stored in the training data storage unit 2660, and extracts the components corresponding to the read component-specific scores from the prompt.

[0218] The weight calculation unit 3810 calculates the sum (total score) of the scores of each component element for each learning data item from the scores of each learning data item corresponding to the component elements extracted from the prompt.

[0219] The weight calculation unit 3810 compares the total scores of each training data and selects the training data with the highest total score to designate the corresponding LoRA as the active LoRA. The weight calculation unit 3810 notifies the prompt generation unit 340 of multiple combinations of weights for each designated active LoRA.

[0220] This allows the generation AI 111 to operate using each execution LoRA under multiple weight combinations. Images corresponding to each combination generated by the generation AI 111 can be displayed to the user.

[0221] As a result, the user evaluates each image, and the weight calculation unit 3810 receives the evaluation results from the user. The weight calculation unit 3810 stores in the weight storage unit 3820 the combination of weights that has been most highly evaluated by the user from among the received evaluation results.

[0222] <Specific example of processing by weight calculation unit> A specific example of processing by the weight calculation unit 3810 of the information processing device 120 according to the seventh embodiment will be described. Fig. 39 is a third diagram showing a specific example of processing by the weight calculation unit. Differences from the first diagram shown in Fig. 33 are reference numerals 3921, 3922, 3940, the weight calculation unit 3810, and the weight storage unit 3820.

[0223] As indicated by reference numeral 3921, in the information processing device 120 according to the seventh embodiment, the weight calculation unit 3810 notifies the prompt generation unit 340 of multiple combinations of weights for multiple specified execution LoRAs. In response to the notification of the multiple combinations to the prompt generation unit 340, corresponding images are displayed to the user, and the user's evaluation results are input. As a result, as indicated by reference numeral 3922, the weight calculation unit 3810 stores the weight combination with the highest evaluation among the received evaluation results in the weight storage unit 3820.

[0224] In this way, by storing the highly evaluated weight combination for each specified combination of multiple execution LoRAs in the weight storage unit 3820, it becomes possible to use the execution LoRA combination with appropriate weights thereafter.

[0225] 39, the weight calculation unit 3810 is configured to prepare a plurality of predetermined weight combinations and store the weight combination that is most highly rated by the user, thereby optimizing the weight combination. However, the method for optimizing the weight combination is not limited to this. For example, each time the weight combination is changed, the evaluation result for the image based on the weight combination before the change and the evaluation result for the image based on the weight combination after the change may be compared, and the weight combination may be changed in a direction that improves the evaluation result.

[0226] <Flow of image generation process by image generation system> The following describes the flow of image generation processing in the generation phase by the image generation system 3100. Fig. 40 is a fourth flowchart showing the flow of image generation processing by the image generation system.

[0227] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0228] In step S1702, the information processing device 120 generates an environmental language by converting the acquired environmental data into a natural language including sensory expressions.

[0229] In step S1703, the information processing device 120 generates a prompt based on either or both of the generated environment language and the acquired environment data.

[0230] In step S4001, the information processing device 120 extracts components corresponding to the component-specific scores associated with each training data from the sentence included in the generated prompt. The information processing device 120 calculates the sum (total score) of the scores of each component for each training data from the scores corresponding to the extracted components, and designates the execution LoRA.

[0231] In step S4002, the information processing device 120 determines whether the weight combination of the specified execution LoRA is stored in the weight storage unit 3820. If it is determined in step S4002 that the weight combination is stored (YES in step S4002), the process proceeds to step S4003.

[0232] In step S4003, the information processing device 120 transmits a prompt including the designation of the execution LoRA and the weight combination of the designated execution LoRA to the server device 110, and operates the generation AI 111 using the execution LoRA under the weight combination. As a result, the generation AI 111 of the server device 110 generates an image corresponding to the weight combination.

[0233] In step S1705, the information processing device 120 acquires the image generated by the generation AI 111 and displays it to the user.

[0234] On the other hand, if it is determined in step S4002 that the information is not stored (NO in step S4002), the process proceeds to step S4004.

[0235] In step S4004, the information processing device 120 acquires multiple combinations of weights for the specified execution LoRA.

[0236] In step S4005, the information processing device 120 sequentially transmits the specified combinations of weights for the execution LoRA to the server device 110, including them in prompts, and sequentially operates the generation AI using the execution LoRA for each combination. As a result, the generation AI of the server device 110 generates an image according to each combination.

[0237] In step S4006, the information processing device 120 acquires the images corresponding to the respective combinations generated by the generation AI 111 and displays them to the user, allowing the user to evaluate each image.

[0238] In step S4007, the information processing device 120 stores the combination of weights corresponding to the image with the highest evaluation result input by the user in the weight storage unit 3820. Furthermore, the information processing device 120 displays the image with the highest evaluation result input by the user on the display screen.

[0239] In step S1706, the information processing device 120 determines whether or not to end the image generation process. If it is determined in step S1706 that the image generation process is to be continued (NO in step S1706), the process returns to step S1701.

[0240] On the other hand, if it is determined in step S1706 that the image generation process is to be ended (YES in step S1706), the image generation process is ended.

[0241] <Summary> As is clear from the above description, the information processing device 120 according to the seventh embodiment: - Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. ·Generating prompts that include phrases or sentences or images that express images that co-occur from environmental language or environmental data. · Specify the LoRA to be performed based on the sentence components contained in the generated prompt. - Include multiple combinations of specified execution LoRA weights in the prompt to operate the generation AI. The user evaluates multiple images generated by the generation AI, and the weight combination corresponding to the image with the highest evaluation is stored in the weight storage unit.

[0242] In this way, the information processing device 120 according to the seventh embodiment uses a weighting combination highly rated by users when operating a generation AI that has undergone additional learning using learning data generated for each type of image content. As a result, the information processing device 120 according to the seventh embodiment can generate images that intuitively represent the state of air with higher accuracy.

[0243] [Eighth embodiment] In the fourth to seventh embodiments, the tuned model 3110 of the server device 110 has been described as having a plurality of arbitrary LoRAs. In contrast, in the eighth embodiment, the plurality of LoRAs are categorized into configuration LoRAs, element LoRAs, and environment LoRAs, and the generation AI is operated sequentially using LoRAs of each category. The eighth embodiment will be described below, focusing on the differences from the fourth to seventh embodiments.

[0244] <System configuration of image generation system> The following describes the system configuration in the learning phase of an image generation system to which an information processing device according to the eighth embodiment is applied. Fig. 41 is a fourth diagram showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the learning phase.

[0245] 41, the image generation system 4100 includes a server device 110, an information processing device 120, and a server device 2540. In the image generation system 4100, the information processing device 120, the server device 110, and the server device 2540 are communicably connected via a network 150.

[0246] In the learning phase, the server device 110 has a generation AI 111, a configuration fine-tuning unit 4111, an element fine-tuning unit 4112, and an environment fine-tuning unit 4113. When the server device 110 receives configuration training data (an example of first training data) from the information processing device 120 via the network 150, the server device 110 uses the configuration fine-tuning unit 4111 to perform additional training on the generation AI 111. As a result, the configuration fine-tuning unit 4111 generates a configuration-tuned model corresponding to each piece of configuration training data.

[0247] Similarly, when the server device 110 receives element training data (an example of second training data) from the information processing device 120 via the network 150, it uses the element fine-tuning unit 4112 to perform additional training on the generation AI 111. As a result, the element fine-tuning unit 4112 generates an element-tuned model corresponding to each element training data.

[0248] Similarly, when the server device 110 receives environment learning data (an example of third learning data) from the information processing device 120 via the network 150, it uses the environment fine-tuning unit 4113 to perform additional learning on the generation AI 111. As a result, the environment fine-tuning unit 4113 generates an environment-tuned model corresponding to each environment learning data.

[0249] In the learning phase, the information processing device 120 generates learning data and instructs the generation AI 111 to perform additional learning. Specifically, the information processing device 120 acquires images to be included in the configuration learning data from the server device 2540. The information processing device 120 generates configuration learning data that includes the acquired images and captions indicating each component included in the acquired images. The information processing device 120 transmits the generated configuration learning data to the server device 110 and instructs the generation AI 111 to perform additional learning using the generated configuration learning data.

[0250] Similarly, the information processing device 120 acquires images to be included in the element training data from the server device 2540. The information processing device 120 generates element training data including the acquired images and captions indicating each component included in the acquired images. The information processing device 120 transmits the generated element training data to the server device 110 and instructs the server device 110 to perform additional training on the generated AI 111 using the generated element training data.

[0251] Similarly, the information processing device 120 acquires images to be included in the environment learning data from the server device 2540. The information processing device 120 generates environment learning data including the acquired images and captions indicating each component included in the acquired images. The information processing device 120 transmits the generated environment learning data to the server device 110 and instructs the server device 110 to perform additional learning on the generated AI 111 using the generated environment learning data.

[0252] In the learning phase, the information processing device 120 calculates a component-specific score for each learning data (each configuration learning data, each element learning data, each environment learning data) based on a score indicating the importance of each component within the image, and stores the score in association with the learning data.

[0253] The server device 2540 functions as an information providing unit 2541 and provides the images to be included in the configuration learning data, element learning data, and environment learning data to the information processing device 120 via the network 150.

[0254] <Examples of learning data> A description will be given of specific examples of learning data for each category (here, configuration learning data, element learning data, and environment learning data) generated by the information processing device 120 in the learning phase. Fig. 42 is a diagram showing specific examples of learning data for each category.

[0255] As shown in FIG. 42, the configured learning data 4210 includes learning data (for sunsets), learning data (for mountains), learning data (for beaches), learning data (for parks), and the like.

[0256] The element training data 4220 includes training data (for clouds), training data (for trees), training data (for flowers), training data (for rivers), and the like.

[0257] The environment learning data 4230 includes learning data (for temperature), learning data (for humidity), learning data (for wind speed), learning data (for rainfall), and the like.

[0258] The items included in each training data are the same regardless of the type of training data, and include "image data" and "caption" as described with reference to Fig. 28. Furthermore, each training data is associated with a "score by component element".

[0259] <System configuration of image generation system> The system configuration in the generation phase of an image generation system to which an information processing device according to the eighth embodiment is applied will be described. Fig. 43 is a fifth diagram showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the generation phase.

[0260] 43, an image generation system 4300 includes a server device 110, an information processing device 120, and a server device 140. In the image generation system 4300, the information processing device 120, the server device 110, and the server device 140 are communicably connected via a network 150.

[0261] In the generation phase, the server device 110 has a generation AI 111 and tuned models for each category (a configuration-tuned model 4311, an element-tuned model 4312, and an environment-tuned model 4313). When the server device 110 receives a prompt from the information processing device 120 via the network 150, it operates the generation AI 111 using the configuration-tuned model 4311. As a result, the server device 110 generates a first image as an image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0262] When the server device 110 receives a prompt including a first image from the information processing device 120 via the network 150, it operates the generation AI 111 using the element-tuned model 4312. As a result, the server device 110 generates a second image as an image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0263] When the server device 110 receives a prompt including the second image from the information processing device 120 via the network 150, it operates the generated AI 111 using the environment-tuned model 4313. As a result, the server device 110 generates a third image as an image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0264] The information processing device 120 is the same as the information processing device 120 described in the fourth embodiment with reference to Fig. 31, and therefore a description thereof will be omitted here. However, in the case of the information processing device 120 according to the eighth embodiment, when transmitting a prompt to the server device 110, a prompt to generate a first image by operating the generation AI 111 using the configured tuned model 4311; a prompt including a first image, the prompt instructing the generation AI 111 to generate a second image by using the element-tuned model 4312; and a prompt including a second image, the prompt instructing the generation AI 111 to generate a third image by using the environment-tuned model; and are transmitted sequentially.

[0265] <Flow of image generation process by image generation system> The following describes the flow of image generation processing in the generation phase by the image generation system 4300. Fig. 44 is a fifth flowchart showing the flow of image generation processing by the image generation system. Note that the difference from the first flowchart shown in Fig. 17 is steps S4401 to S4404.

[0266] In step S4401, the information processing device 120 extracts components corresponding to the component-specific scores associated with each piece of training data from the sentence included in the generated prompt. The information processing device 120 calculates the sum (total score) of the scores of each component for each piece of training data from the scores corresponding to the extracted components. As a result, the information processing device 120 calculates the following based on the total score of each piece of training data: ·Specify the execution LoRA of the configured tuned model 4311. ·Specify the execution LoRA of the element-tuned model 4312. ·Specify the execution LoRA of the environmentally tuned model 4313.

[0267] In step S4402, the information processing device 120 sends a prompt including a designation of the execution LoRA of the configuration-tuned model 4311 to the server device 110. As a result, the server device 110 operates the generation AI using the execution LoRA of the configuration-tuned model 4311 to generate a first image.

[0268] In step S4403, the information processing device 120 transmits to the server device 110 a prompt including the generated first image and a designation of the execution LoRA of the element-tuned model 4312. As a result, the server device 110 operates the generation AI using the execution LoRA of the element-tuned model 4312 to generate the second image.

[0269] In step S4404, the information processing device 120 transmits to the server device 110 a prompt including the generated second image and a designation of the execution LoRA of the environmentally tuned model 4313. As a result, the server device 110 operates the generation AI using the execution LoRA of the environmentally tuned model 4313 to generate the third image.

[0270] <Summary> As is clear from the above description, the information processing device 120 according to the eighth embodiment performs the following in the learning phase: - Recognize each component contained in the collected images and calculate the reliability of each recognized component. The collected images are classified into multiple groups according to the type of image content, and scores for each component element contained in the images belonging to each group are calculated. For each group, training data is generated in which images are associated with captions indicating each component element contained in the image, and a component-specific score is associated with each training data. · The generated learning data is categorized into configuration learning data, element learning data, and environment learning data. Using the configuration learning data, the corresponding LoRA in the configuration fine-tuning unit is used to perform additional training on the generation AI, thereby generating a configuration-tuned model. Using the element learning data, the corresponding LoRA in the element fine-tuning section is used to perform additional learning on the generation AI, thereby generating an element-tuned model. Using the environmental learning data, the corresponding LoRA in the environmental fine-tuning unit is used to perform additional training on the generation AI, thereby generating an environmentally tuned model.

[0271] Furthermore, the information processing apparatus 120 according to the eighth embodiment performs the following in the generation phase: - Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. ·Generating prompts that include phrases or sentences or images that express images that co-occur from environmental language or environmental data. · Designate an execution LoRA for each category based on the sentence components contained in the generated prompt. Send a prompt specifying the LoRA to run the configured tuned model. This will run the generation AI using the LoRA of the configured tuned model and acquire the first image. Send a prompt including the LoRA specification for the element-tuned model and the first image acquired, which will run the generative AI using the LoRA for the element-tuned model and acquire the second image. Send a prompt specifying the LoRA to run the environmentally tuned model and the second image you captured. This will run the generative AI using the LoRA of the environmentally tuned model and capture the third image. The acquired third image is captured as an image showing the air quality of the target environment and displayed to the user.

[0272] In this way, in the information processing device 120 according to the eighth embodiment, the generation AI that has undergone additional learning for each category (configuration, element, environment) is operated in the order of the categories (configuration, element, environment). As a result, the information processing device 120 according to the eighth embodiment can more appropriately generate an image that intuitively expresses the state of the air.

[0273] In the above description, configuration, element, and environment are listed as categories, but the number and type of categories are not limited to these. Furthermore, in the above description, one execution LoRA is designated for each category, but the number of execution LoRAs designated for each category is not limited to one and may be multiple. For example, as described in the sixth embodiment, multiple execution LoRAs may be designated for a particular category as a result of redesignating the execution LoRA when it is determined that the sentence included in the prompt does not satisfy the environmental conditions assigned to the training data.

[0274] [Ninth embodiment] A specific example of an image that intuitively expresses the state of air, generated by an image generation system including the information processing device 120 according to the sixth and eighth embodiments, will be described. Fig. 45 is a diagram showing an example of image generation processing by the image generation system.

[0275] In the example of FIG. 45, the information processing device 120 sets the following as environmental data: Date and time: July 25th, 4:00 PM Location: Karuizawa ·Temperature: 25℃ ·Humidity: 60% ·Air volume: 2m / s The figure shows how the above information was acquired.

[0276] The example in FIG. 45 shows how the information processing device 120 generates environmental terms such as "hot," "relatively clear visibility," "slightly strong wind," "sunset," and "summer resort" based on environmental data.

[0277] The example in Figure 45 shows how the information processing device 120 generates, based on the environmental language and environmental data, the following sentence to be included in the prompt: "A summer evening in Karuizawa. Under the intense sunlight, you can hear the chirping of cicadas. When you enter the shade of the trees, a refreshing breeze blows through, providing a little relief from the heat."

[0278] In the example of FIG. 45, the information processing device 120 selects the following as image editing parameters to be included in the prompt based on the environmental data: Brightness: +15, Contrast:+10, Saturation: +15, Sharpness: +10, Blur: 2~3, This shows how the above was identified.

[0279] In the example of FIG. 45, the information processing device 120: Specify the LoRA for sunset as the execution LoRA for the configured tuned model, Instead of the execution LoRA of the environmentally tuned model, specify the LoRA for wind speed of the element-tuned model. -Specify the temperature LoRA as the execution LoRA of the element-tuned model. It shows the situation.

[0280] In the example of Figure 45, reference numerals 4501 to 4503 indicate, for reference, images acquired when the information processing device 120 operates the generation AI by separately including the execution LoRA for each specified category in the prompt.

[0281] On the other hand, in the example of Figure 45, reference numeral 4510 indicates an image acquired when the information processing device 120 operates the generation AI by sequentially including the specification of the execution LoRA for each category in the prompt. As is clear from the comparison between the images indicated by reference numerals 4501 to 4503 and the image indicated by reference numeral 4510, -When generating images by running each category of LoRA individually, -When images are generated by running each category of LoRA sequentially, The images produced are very different.

[0282] In this way, by categorizing multiple tuned models and running them in the appropriate order, it is possible to generate highly accurate images that intuitively represent the air condition.

[0283] [Other embodiments] In the above embodiments, the information processing program is executed by one information processing device 120, but the information processing program may be executed by a plurality of information processing devices in cooperation with each other.

[0284] Note that the functions realized by the information processing device 120 by executing the information processing program are not limited to the functions described in the above embodiments. For example, some or all of the functions of the server device 110 and some or all of the functions of the server device 140 may be realized in the information processing device 120.

[0285] Although the embodiments have been described above, it will be understood that various changes in form and details can be made without departing from the spirit and scope of the claims. [Explanation of symbols]

[0286] 100: Image generation system 110: Server device 111: Generation AI 120: Information processing device 130_1~130_n: Sensor 1~Sensor n 200: Control section 310: Communication control unit 320: Environmental data acquisition unit 330:Environmental language generation section 340: Prompt generation unit 350: Output section 2110: Motion control language generation unit 2500: Image generation system 2511: Fine Tuning Section 2540: Server device 2610: Image Data Collection Department 2620: Caption generation unit 2630: Learning data generation unit 2640: Additional Learning Section 3100: Image generation system 3110: Tuned model 3210: Weight calculation unit 3610: Selection section 3810: Weight calculation unit 4100: Image generation system 4111: Configuration fine tuning section 4112: Element Fine Tuning Unit 4113: Environmental Fine Tuning Department 4210: Data for construction learning 4220: Element learning data 4230: Environmental learning data 4300: Image generation system 4311: Configuration tuned model 4312: Element-tuned model 4313: Environmentally tuned model

Claims

1. An information processing device having a control unit, The control unit Environmental data of the target environment is acquired and converted into natural language including sensory expressions to generate environmental language. generating prompts including phrases or sentences that represent images that co-occur from the ambient language; Each component is extracted from the generated prompt, and the score of each extracted component is calculated for each training data based on the scores for each component associated with the training data, thereby calculating a total score for each training data; Selecting training data based on the total score, and operating a generated AI that has been additionally trained using a model corresponding to the selected training data based on the weight of the model and the prompt; Displaying an image showing the air condition of the target environment generated by the generation AI. Information processing device.

2. The generating AI is additionally trained using a model for generating a specific image and corresponding learning data, The corresponding learning data includes a group of specific images and captions indicating each component included in each specific image. The information processing device according to claim 1 .

3. The learning data corresponding to the model for generating the specific image is associated with a score for each component element according to the reliability of each component element calculated when recognizing each component element included in each specific image. The information processing device according to claim 2 .

4. The control unit When operating the generation AI that has been additionally trained using a model corresponding to the selected learning data, a weight of the corresponding model is calculated based on the total score. The information processing device according to claim 1 .

5. The control unit Using a large-scale language model, determine the importance of the model corresponding to the selected training data for the prompt; Calculating a weight of the model when operating the generation AI that has been additionally trained using a model corresponding to the selected learning data based on the determined importance; The information processing device according to claim 1 .

6. The control unit determining an environmental condition assigned to the selected learning data; The information processing device according to claim 1 .

7. The control unit If an environmental condition is assigned to the selected training data and the sentence included in the prompt satisfies the environmental condition, a generation AI that has been additionally trained using a model corresponding to the selected training data is operated under the weight of the model; If an environmental condition is assigned to the selected learning data and the sentence included in the prompt does not satisfy the environmental condition, a generation AI that has been additionally trained using another model other than the model corresponding to the selected learning data is operated under the weight of the other model; If no environmental conditions are assigned to the selected learning data, a generation AI additionally trained using a model corresponding to any learning data is operated under the weights of the model. The information processing device according to claim 6 .

8. The control unit Obtaining a plurality of combinations of weights for the model when operating the generation AI that has been additionally trained using a model corresponding to the selected learning data; The generation AI, which has been additionally trained using a model corresponding to the selected learning data, is operated under a combination of weights of each model, thereby identifying a combination of weights of a model corresponding to an image evaluated by a user from among the images generated by the generation AI. The information processing device according to claim 1 .

9. The learning data is First learning data for additionally learning the image configuration; Second learning data for additionally learning predetermined elements included in the image; Third learning data for additionally learning the environment represented by the image; Including, The control unit By operating a generation AI that has been additionally trained using the first learning data, a first image generated by the generation AI is obtained; A generation AI that has been additionally trained using the second learning data is operated using the first image to obtain a second image generated by the generation AI; A generation AI that has been additionally trained using the third learning data is operated using the second image to obtain a third image generated by the generation AI; displaying the acquired third image as an image showing the air condition of the target environment; The information processing device according to claim 1 .

10. The control unit: Recognizing each component included in each specific image; Using learning data including a group of specific images and captions indicating each component recognized in each specific image, additional training of the generation AI is performed using a model for generating specific images; a score for each component element according to the reliability of each component element calculated when recognizing each component element included in each specific image, the score being associated with the learning data and stored; The information processing device according to claim 1 .

11. The learning data is First learning data for additionally learning the image configuration; Second learning data for additionally learning predetermined elements included in the image; Third learning data for additionally learning the environment represented by the image; Including, The information processing device according to claim 10.

12. A control unit included in the information processing device Environmental data of the target environment is acquired and converted into natural language including sensory expressions to generate environmental language. generating prompts including phrases or sentences that represent images that co-occur from the ambient language; Each component is extracted from the generated prompt, and the score of each extracted component is calculated for each training data based on the scores for each component associated with the training data, thereby calculating a total score for each training data; Selecting training data based on the total score, and operating a generated AI that has been additionally trained using a model corresponding to the selected training data based on the weight of the model and the prompt; Displaying an image showing the air condition of the target environment generated by the generation AI. An information processing method that performs processing.

13. A control unit included in the information processing device Recognizing each component included in each specific image; Using learning data including a group of specific images and captions indicating each component included in each specific image, additional training of the generation AI is performed using a model for generating specific images, a score for each component element according to the reliability of each component element calculated when recognizing each component element included in each specific image, the score being associated with the learning data and stored; The information processing method according to claim 12, further comprising: executing a process.

14. A control unit of the information processing device includes: Environmental data of the target environment is acquired and converted into natural language including sensory expressions to generate environmental language. generating prompts including phrases or sentences that represent images that co-occur from the ambient language; Each component is extracted from the generated prompt, and the score of each extracted component is calculated for each training data based on the scores for each component associated with the training data, thereby calculating a total score for each training data; Selecting training data based on the total score, and operating a generated AI that has been additionally trained using a model corresponding to the selected training data based on the weight of the model and the prompt; Displaying an image showing the air condition of the target environment generated by the generation AI. An information processing program for executing processing.

15. A control unit of the information processing device includes: Recognizing each component included in each specific image; Using learning data including a group of specific images and captions indicating each component included in each specific image, additional training of the generation AI is performed using a model for generating specific images, a score for each component element according to the reliability of each component element calculated when recognizing each component element included in each specific image, the score being associated with the learning data and stored; The information processing program according to claim 14, which causes a process to be executed.

Images