Information processing device, information processing method, and information processing program

Abstract

It generates images that intuitively represent the state of the air. [Solution] An information processing device having a control unit, wherein the control unit acquires environmental data of a target environment, converts it into natural language including sensory expressions to generate an environmental language, generates a prompt including words or sentences that represent images co-occurring from the environmental language, selects a generation AI that has been previously trained using each of a plurality of training data based on the generated prompt, operates it based on the assigned weights, and displays an image showing the state of the air in the target environment generated by the generation AI.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Techniques for visualizing environmental data such as temperature, humidity, wind speed, and carbon dioxide concentration are known. According to this technique, for example, as the state of air, it is possible to quantitatively grasp the high or low temperature, high or low humidity, large or small wind speed, high or low carbon dioxide concentration, etc.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, there has been no means for sensuously expressing the state of air (for example, means for imaging human senses) until now. For example, even using generative AI or the like, it has been difficult to directly generate an image that sensuously expresses the state of air from the above quantitative data.

[0005] An object of the present disclosure is to generate an image that sensuously expresses the state of air.

Means for Solving the Problems

[0006] A first aspect of the present disclosure is an information processing apparatus having a control unit, wherein the control unit acquires environmental data of a target environment, generates an environmental language by converting it into a natural language including a sensuous expression, generates a prompt including a phrase or sentence expressing an image co-occurring from the environmental language, Based on the generated prompt, select a generator AI that has been pre-trained using multiple training datasets, and operate it based on the assigned weights. The system displays an image showing the air quality of the target environment, generated by the aforementioned AI.

[0007] A second aspect of this disclosure is an information processing apparatus described in the first aspect, The aforementioned generating AI is further trained using a model for generating specific images and corresponding training data. The corresponding training data includes a specific set of images and captions indicating each component contained in each specific image.

[0008] A third aspect of this disclosure is an information processing apparatus described in the second aspect, The training data corresponding to the model for generating the aforementioned specific images is associated with component-specific scores calculated when recognizing each component contained in each of the aforementioned specific images, based on the confidence level of each component.

[0009] A fourth aspect of this disclosure is an information processing apparatus described in any of the first to third aspects, The control unit, The system extracts each component from the generated prompt, and then calculates the total score for each training dataset by summing the scores of each extracted component based on the component-specific scores associated with the training data.

[0010] A fifth aspect of this disclosure is an information processing apparatus as described in the fourth aspect, The control unit, Based on the aforementioned total score, select the training data. When running the generative AI that has been further trained using the model corresponding to the selected training data, the weights of the corresponding model are calculated based on the total score.

[0011] A sixth aspect of this disclosure is an information processing apparatus as described in the fourth aspect, The control unit, Based on the aforementioned total score, select the training data. Using a large-scale language model, the importance of the model's prompts corresponding to the selected training data is determined. Based on the determined importance, the model weights are calculated for when the generative AI, which has been further trained using the model corresponding to the selected training data, is operated.

[0012] A seventh aspect of this disclosure is an information processing apparatus as described in the fourth aspect, The control unit, Based on the aforementioned total score, select the training data. The environmental conditions attached to the selected training data are determined.

[0013] The eighth aspect of this disclosure is an information processing apparatus as described in the seventh aspect, The control unit, If environmental conditions are attached to the selected training data, and the sentence included in the prompt satisfies the environmental conditions, the generative AI, which has been further trained using the model corresponding to the selected training data, is operated based on predetermined weights. If environmental conditions are attached to the selected training data, and the sentence included in the prompt does not satisfy the environmental conditions, then a generative AI that has been further trained using a model other than the model corresponding to the selected training data is operated based on predetermined weights. If no environmental conditions are attached to the selected training data, a generative AI, further trained using a model corresponding to arbitrary training data, is made to operate based on predetermined weights.

[0014] A ninth aspect of this disclosure is an information processing apparatus as described in the fourth aspect, The control unit, Based on the aforementioned total score, select the training data. When operating the generative AI additionally learned using the model corresponding to the selected learning data, a plurality of combinations of weights of the model are obtained. By operating the generative AI additionally learned using the model corresponding to the selected learning data based on each combination of weights, among the images generated by the generative AI, the combination of weights corresponding to the image evaluated by the user is specified.

[0015] A tenth aspect of the present disclosure is an information processing apparatus according to any one of the first to ninth aspects, where the learning data includes first learning data for additionally learning the configuration of an image, second learning data for additionally learning a predetermined element included in the image, third learning data for additionally learning the environment represented by the image, and includes where the control unit obtains a first image generated by the generative AI by operating the generative AI additionally learned using the first learning data, obtains a second image generated by the generative AI by operating the generative AI additionally learned using the second learning data using the first image, obtains a third image generated by the generative AI by operating the generative AI additionally learned using the third learning data using the second image, and displays the obtained third image as an image showing the state of the air in the target environment.

[0016] An eleventh aspect of the present disclosure is an information processing apparatus having a control unit, where the control unit recognizes each component included in each specific image, and additionally learns the generative AI using a model for generating a specific image using learning data including a group of specific images and captions indicating each component recognized in each specific image. The component-specific scores, calculated when recognizing each component contained in each of the aforementioned specific images, according to the confidence level of each component, are stored in association with the training data.

[0017] A twelfth aspect of this disclosure is an information processing apparatus described in the eleventh aspect, The aforementioned training data is The first training data for further learning the image composition, Second training data for further learning of specific elements contained in the image, This includes a third set of training data to further train the model on the environment represented by the image.

[0018] A thirteenth aspect of this disclosure is an information processing method, The control unit of the information processing device, By acquiring environmental data from the target environment and converting it into natural language that includes sensory expressions, an environmental language is generated. A prompt is generated that includes words or sentences representing images that co-occur from the aforementioned environmental language. Based on the generated prompt, select a generator AI that has been pre-trained using multiple training datasets, and operate it based on the assigned weights. The process is executed to display an image showing the air quality of the target environment, which was generated by the aforementioned AI.

[0019] A fourteenth aspect of this disclosure is an information processing method, The control unit of the information processing device, Recognize each component contained in each specific image, Using training data that includes a specific set of images and captions describing each component contained in each specific image, the generative AI is further trained using a model for generating specific images. The process involves storing the component-specific scores, calculated when recognizing each component in each of the aforementioned specific images, in association with the training data, based on the confidence level of each component.

[0020] A 15th aspect of this disclosure is an information processing program, The control unit of the information processing device has By acquiring environmental data from the target environment and converting it into natural language that includes sensory expressions, an environmental language is generated. A prompt is generated that includes words or sentences representing images that co-occur from the aforementioned environmental language. Based on the generated prompt, select a generator AI that has been pre-trained using multiple training datasets, and operate it based on the assigned weights. The system is configured to execute a process that displays an image representing the air quality of the target environment, which was generated by the aforementioned AI.

[0021] A sixteenth aspect of this disclosure is an information processing program, The control unit of the information processing device has Recognize each component contained in each specific image, Using training data that includes a specific set of images and captions describing each component contained in each specific image, the generative AI is further trained using a model for generating specific images. The system executes a process that stores, in association with the training data, the component-specific scores calculated when recognizing each component contained in each of the aforementioned specific images, according to the confidence level of each component. [Brief explanation of the drawing]

[0022] [Figure 1] The first figure shows an example of the system configuration of an image generation system. [Figure 2] This figure shows an example of the hardware configuration of an information processing device. [Figure 3] The first figure shows an example of the functional configuration of an information processing device. [Figure 4] This figure shows a specific example of processing performed by the environmental data acquisition unit. [Figure 5] Figure 1 shows a specific example of processing by the environment language generation unit. [Figure 6] The second figure shows a specific example of processing by the environment language generation unit. [Figure 7] The third figure shows a specific example of processing by the environment language generation unit. [Figure 8] The first figure shows a specific example of processing by the prompt generation unit. [Figure 9] The second figure shows a specific example of processing by the prompt generation unit. [Figure 10] The third figure shows a specific example of processing by the prompt generation unit. [Figure 11] Figure 4 shows a specific example of processing by the prompt generation unit. [Figure 12] Figure 5 shows a specific example of processing by the prompt generation unit. [Figure 13] Figure 6 shows a specific example of processing by the prompt generation unit. [Figure 14] Figure 7 shows a specific example of processing by the prompt generation unit. [Figure 15] Figure 8 shows a specific example of processing by the prompt generation unit. [Figure 16] Figure 9 shows a specific example of processing by the prompt generation unit. [Figure 17] This is the first flowchart illustrating the image generation process by the image generation system. [Figure 18] Figure 1 shows an example of the display screen. [Figure 19] The second figure shows an example of the functional configuration of an information processing device. [Figure 20] The second figure shows an example of the display screen. [Figure 21] This is the third figure, which shows an example of the functional configuration of an information processing device. [Figure 22] This figure shows a specific example of processing performed by the operation control language generation unit. [Figure 23] This figure shows an example of a prompt (text) and an example of a video. [Figure 24] This is the second flowchart illustrating the image generation process by the image generation system. [Figure 25]The second figure shows an example of the system configuration of an image generation system. [Figure 26] Figure 4 shows an example of the functional configuration of an information processing device. [Figure 27] This figure shows a specific example of processing performed by the caption generation unit. [Figure 28] This figure shows a specific example of processing performed by the training data generation unit. [Figure 29] This diagram shows the details of the generation AI and fine-tuning section. [Figure 30] This flowchart shows the flow of additional learning processing by the image generation system. [Figure 31] The third figure shows an example of the system configuration of an image generation system. [Figure 32] Figure 5 shows an example of the functional configuration of an information processing device. [Figure 33] The first figure shows a specific example of processing by the weight calculation unit. [Figure 34] This is the third flowchart illustrating the image generation process by the image generation system. [Figure 35] The second figure shows a specific example of processing by the weight calculation unit. [Figure 36] Figure 6 shows an example of the functional configuration of an information processing device. [Figure 37] This figure shows a specific example of processing performed by the selection unit. [Figure 38] Figure 7 shows an example of the functional configuration of an information processing device. [Figure 39] The third figure shows a specific example of processing by the weight calculation unit. [Figure 40] This is the fourth flowchart illustrating the image generation process by the image generation system. [Figure 41] Figure 4 shows an example of the system configuration of an image generation system. [Figure 42] This figure shows specific examples of training data for each category. [Figure 43] Figure 5 shows an example of the system configuration of an image generation system. [Figure 44] This is the fifth flowchart illustrating the image generation process by the image generation system. [Figure 45] This figure shows an example of image generation processing by an image generation system. [Modes for carrying out the invention]

[0023] Each embodiment will be described below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals to avoid redundant descriptions.

[0024] [First Embodiment] <System configuration of the image generation system> The system configuration of an image generation system to which the information processing device according to the first embodiment is applied will now be described. Figure 1 is a first diagram showing an example of the system configuration of the image generation system.

[0025] As shown in Figure 1, the image generation system 100 comprises a server device 110, an information processing device 120, and a server device 140. In the image generation system 100, the information processing device 120 and the server devices 110 and 140 are connected to each other via a network 150.

[0026] The server device 110 has a generation AI 111, and when it receives a prompt from the information processing device 120 via the network 150, it operates the generation AI 111 to generate an image (still image or moving image) corresponding to the prompt. The server device 110 transmits the generated image to the information processing device 120 via the network 150.

[0027] The information processing device 120 is a device that displays an image that intuitively represents the state of the air in a target environment. In the first embodiment, the information processing device 120 causes the generation AI 111 to generate an image (still image) that intuitively represents the state of the air in the target environment, and retrieves and displays the image from the generation AI 111. The target environment refers to the space that is the target when representing the state of the air. The information processing device 120 performs various processes necessary to cause the generation AI 111 to generate an image.

[0028] Specifically, the information processing device 120 is: • Environmental data of the target environment measured by sensors 1 to n (symbols 130_1 to 130_n) installed in the target environment (e.g., data such as temperature and humidity), or • Environmental data of the target environment entered by the user (e.g., data such as season and location), The information processing device 120 acquires environmental data of the target environment (for example, rainfall, wind speed, etc.) from the server device 140 via the network 150.

[0029] The information processing device 120 generates an environmental language by converting acquired environmental data of the target environment into natural language that includes sensory expressions. The information processing device 120 generates a prompt that includes words, sentences, or images that represent images that co-occur from either the environmental data or the environmental language, or both. The information processing device 120 uses the generated prompt to operate a generation AI and displays an image generated by the generation AI that shows the air quality of the target environment.

[0030] The server device 140 functions as an information provision unit 141 and provides environmental data of the target environment (e.g., rainfall, wind speed, etc.) to the information processing device 120 via the network 150. For example, the server device 140 acquires location information (latitude, longitude, altitude) of the target environment from the information processing device 120 and collects environmental data corresponding to the location information at predetermined intervals. The server device 140 provides the collected environmental data to the information processing device 120 at predetermined intervals.

[0031] Although Figure 1 shows an example where the information processing device 120 is installed outside the target environment, the information processing device 120 may also be installed inside the target environment. Furthermore, Figure 1 does not mention the user who views the image showing the air quality of the target environment displayed by the information processing device 120, but the user may view the image inside or outside the target environment.

[0032] <Hardware configuration of the information processing device> The hardware configuration of the information processing device 120 will now be described. Figure 2 shows an example of the hardware configuration of the information processing device. As shown in Figure 2, the information processing device 120 includes a processor 201, memory 202, auxiliary storage device 203, user interface device 204, communication device 205, connection device 206, and drive device 207. Each piece of hardware in the information processing device 120 is interconnected via a bus 208.

[0033] The processor 201 has various computing devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 201 reads various programs (for example, information processing programs, etc.) into memory 202 and executes them.

[0034] Memory 202 has main memory devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The processor 201 and memory 202 form a so-called computer (also called the "control unit 200"), and the computer realizes various functions by having the processor 201 execute various programs read from memory 202.

[0035] The auxiliary storage device 203 stores various programs and various information used when those programs are executed by the processor 201.

[0036] The user interface device 204 includes an operating device for inputting user instructions and a display device for displaying a screen including an image.

[0037] The communication device 205 is connected to the network 150 and is a device that performs communication processing with server devices 110, 140, etc.

[0038] The connection device 206 is a device that connects sensors 1 to n (code 130_1 to 130_n) to the information processing device 120.

[0039] The drive device 207 is a device for setting the recording medium 210. The recording medium 210 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 210 may also include semiconductor memory that records information electrically, such as ROM and flash memory.

[0040] The various programs to be installed on the auxiliary storage device 203 are installed, for example, when the distributed recording medium 210 is set in the drive device 207 and the various programs recorded on the recording medium 210 are read by the drive device 207. Alternatively, the various programs to be installed on the auxiliary storage device 203 may be installed by downloading them from the network 150 via the communication device 205.

[0041] <Functional Configuration of Information Processing Devices> The functional configuration of the information processing device 120 will now be described. Figure 3 is the first diagram showing an example of the functional configuration of the information processing device. As described above, an information processing program is installed in the information processing device 120, and when this information processing program is executed, the information processing device 120 operates as follows: • Communication control unit 310, • Environmental data acquisition unit 320, ·Environmental language generation unit 330, • Prompt generation unit 340, Output section 350, It functions as such.

[0042] The communication control unit 310 acquires environmental data from the server device 140 via the network 150 and notifies the environmental data acquisition unit 320. The communication control unit 310 acquires the prompt generated by the prompt generation unit 340 and transmits it to the server device 110 via the network 150, and also receives an image from the server device 110 and notifies the output unit 350.

[0043] The environmental data acquisition unit 320 is • Environmental data notified from the communication control unit 310, • Environmental data measured by sensors 1 to n (codes 130_1 to 130_n) • User-entered environmental data, The data is obtained and notified to the environment language generation unit 330 and the prompt generation unit 340.

[0044] The environmental language generation unit 330 generates environmental language based on environmental data notified by the environmental data acquisition unit 320 by referring to the conversion table stored in the conversion table storage unit 360. Environmental language is a natural language that expresses the state of the air in the target environment in a sensory way. "Expressing in a sensory way" means putting into words the sensations that a person actually in the target environment would feel. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0045] The prompt generation unit 340 is, - A word, sentence, or image that represents an image that co-occurs from either or both of the environmental data notified by the environmental data acquisition unit 320 and the environmental language notified by the environmental language generation unit 330, • Image editing parameters corresponding to the environmental data notified from the environmental data acquisition unit 320, The prompt generation unit 340 generates a prompt that includes the specified parameters. The prompt generation unit 340 notifies the communication control unit 310 of the generated prompt and also notifies the output unit 350. The image editing parameters refer to quantitative indicators that affect image quality, and specific examples will be described later.

[0046] The output unit 350 generates a display screen based on the image notified by the communication control unit 310 and the words or sentences, image editing parameters, etc., included in the prompt notified by the prompt generation unit 340, and displays it to the user.

[0047] <Specific examples of processing performed by each part of the information processing device> Specific examples of processing performed by each part of the information processing device 120 (here, the environment data acquisition unit 320, the environment language generation unit 330, and the prompt generation unit 340) will be described.

[0048] (1) Specific example of processing by the environmental data acquisition unit Figure 4 shows a specific example of processing by the environmental data acquisition unit. In Figure 4, reference numeral 401 indicates an example of environmental data provided by the server device 140 and acquired by the environmental data acquisition unit 320 from the communication control unit 310. As shown by reference numeral 401 in Figure 4, the environmental data provided by the server device 140 includes rainfall, wind direction, wind speed, weather, discomfort index, etc.

[0049] In Figure 4, reference numeral 402 indicates an example of environmental data acquired by the environmental data acquisition unit 320 through measurements taken by sensors 1 to n (reference numerals 130_1 to 130_n) or through input by the user. As shown by reference numeral 402 in Figure 4, the environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n) or input by the user includes temperature, humidity, date, time, carbon dioxide concentration, season, location, etc.

[0050] The environmental data acquisition unit 320 notifies the environmental language generation unit 330 of the acquired environmental data. When notifying the environmental data, the environmental data acquisition unit 320 may divide the items included in the environmental data into items that indicate the air quality of the target environment and items that affect the air quality of the target environment.

[0051] The example in Figure 4 shows how the environmental language generation unit 330 is notified of items indicating the air quality of the target environment, such as temperature, humidity, discomfort index, rainfall, wind direction, wind speed, and carbon dioxide concentration. The example in Figure 4 also shows how the environmental language generation unit 330 is notified of items that affect the air quality of the target environment, such as date, time, season, weather, and location. As shown in Figure 4, the item values ​​for each item indicating the air quality of the target environment and the item values ​​for each item that affects the air quality of the target environment may be notified to the environmental language generation unit 330 at predetermined intervals, for example.

[0052] (2) Specific examples of processing by the environment language generation unit Figure 5 is the first diagram showing a specific example of processing by the environmental language generation unit. As shown in Figure 5, the conversion table storage unit 360 stores a conversion table in which the item values ​​of multiple items indicating the air condition of the target environment from the environmental data are associated with environmental language, which is natural language that includes sensory expressions.

[0053] Of these, conversion table 510 is a table that associates each item value of the item "temperature" with environmental language. Conversion table 520 is a table that associates each item value of the item "humidity" with environmental language. Conversion table 530 is a table that associates each item value of the item "discomfort index" with environmental language. Conversion table 540 is a table that associates each item value of the item "rainfall" with environmental language. Conversion table 550 is a table that associates each item value of the item "wind speed" with environmental language. Note that when conversion tables 510 to 550 include item values ​​for items that indicate the air quality of the target environment, By logarithmically transforming each item value, or By approximating each item value with a sigmoid function, It could also be a conversion table that takes into account compatibility with subjective perceptions.

[0054] When the environmental language generation unit 330 receives item values ​​for items indicating the air conditions of the target environment, such as temperature, humidity, discomfort index, rainfall, and wind speed, as environmental data from the environmental data acquisition unit 320, it refers to conversion tables 510 to 550. Based on this, the environmental language generation unit 330 converts the item values ​​for items indicating the air conditions of the target environment, such as temperature, humidity, discomfort index, rainfall, and wind speed, into natural language that includes sensory expressions, and generates environmental language. The environmental language generation unit 330 then notifies the prompt generation unit 340 of the generated environmental language.

[0055] Figure 6 is a second diagram showing a specific example of processing by the environmental language generation unit. As shown in Figure 6, the conversion table storage unit 360 stores a conversion table in which the item values ​​of multiple items that affect the air quality of the target environment are associated with environmental language, which is natural language that includes sensory expressions.

[0056] Of these, conversion table 610 is a table that associates each item value of the item "Date" with the environment language.

[0057] When the environmental language generation unit 330 receives item values ​​of items that affect the air quality of the target environment, such as the date, as environmental data from the environmental data acquisition unit 320, it refers to the conversion table 610. Based on this, the environmental language generation unit 330 converts the item values ​​of items that affect the air quality of the target environment, such as the date, into natural language including sensory expressions, and generates environmental language. The environmental language generation unit 330 then notifies the prompt generation unit 340 of the generated environmental language.

[0058] Figure 7 is a third diagram showing a specific example of processing by the environmental language generation unit. As shown in Figure 7, the conversion table storage unit 360 stores a conversion table in which combinations of item values ​​for multiple items that affect the air quality of the target environment are associated with environmental language, which is natural language that includes sensory expressions.

[0059] Of these, conversion table 710 is a table that associates combinations of item values ​​for the fields "season", "date", and "location" with environment languages. Similarly, conversion table 720 is a table that associates combinations of item values ​​for the fields "season", "date", and "location" with environment languages. Conversion table 710 and conversion table 720 have the same fields, but the item values ​​are different. Therefore, the associated environment languages ​​are different.

[0060] When the environmental language generation unit 330 receives a combination of item values ​​for items that affect the air quality of the target environment, such as season, date, and location, as environmental data from the environmental data acquisition unit 320, it refers to the conversion tables 710-720. The environmental language generation unit 330 then converts the combination of item values ​​for items that affect the air quality of the target environment, such as season, date, and location, into natural language that includes sensory expressions, thereby generating the environmental language. The environmental language generation unit 330 then notifies the prompt generation unit 340 of the generated environmental language.

[0061] (3) Specific example of processing by the prompt generation unit Figure 8 is the first diagram showing a specific example of processing by the prompt generation unit. As described above, the prompt generation unit 340 generates a prompt that includes text representing an image co-occurring from the environmental data notified by the environmental data acquisition unit 320 (in this case, item values ​​of items that affect the air quality of the target environment), and notifies the communication control unit 310.

[0062] The example in Figure 8 shows how the prompt generation unit 340 generates text representing images that co-occur from each item value when the communication control unit 310 notifies the item values ​​of items that affect the air quality of the target environment, such as "date," as environmental data. The prompt generation unit 340 then notifies the communication control unit 310 of the prompt containing the generated text.

[0063] Figure 9 is a second diagram showing a specific example of processing by the prompt generation unit. As described above, the prompt generation unit 340, • Environmental data notified from the environmental data acquisition unit 320 (in this case, item values ​​of items that affect the air quality of the target environment), • The environmental language notified by the environmental language generation unit 330, A prompt containing text describing the image that co-occurs from the input is generated and notified to the communication control unit 310.

[0064] The example in Figure 9 is, The environmental data acquisition unit 320 notifies the item values ​​of items that affect the air quality of the target environment, such as "location," "season," and "time," as environmental data. The environmental language generation unit 330 notified "hot" as the environmental language. In this case, the prompt generation unit 340 generates text that represents an image co-occurring from the environmental data and the environmental language. The prompt generation unit 340 notifies the communication control unit 310 of the prompt containing the generated text.

[0065] Figure 10 is a third diagram showing a specific example of processing by the prompt generation unit. As described above, the prompt generation unit 340, • Environmental data notified from the environmental data acquisition unit 320 (in this case, item values ​​of items that affect the air quality of the target environment), • The environmental language notified by the environmental language generation unit 330, A prompt containing text describing the image that co-occurs from the input is generated and notified to the communication control unit 310.

[0066] The example in Figure 10 is, The environmental data acquisition unit 320 notifies the item values ​​of items that affect the air quality of the target environment, such as "location" and "time," as environmental data. The environmental language generation unit 330 notified "very hot" as the environmental language. In this case, the prompt generation unit 340 generates text that represents an image co-occurring from the environmental data and the environmental language. The prompt generation unit 340 notifies the communication control unit 310 of the prompt containing the generated text.

[0067] Figure 11 is the fourth figure showing a specific example of processing by the prompt generation unit. As described above, the prompt generation unit 340, • Environmental data notified from the environmental data acquisition unit 320 (in this case, item values ​​of items that affect the air quality of the target environment), • The environmental language notified by the environmental language generation unit 330, A prompt containing an image representing the co-occurring image is generated and notified to the communication control unit 310.

[0068] The example in Figure 11 is, The environmental data acquisition unit 320 notifies the item values ​​of items that affect the air quality of the target environment, such as "location" and "time," as environmental data. The environmental language generation unit 330 notified "very hot" as the environmental language. In this case, the prompt generation unit 340 generates an image representing an image that co-occurs from the environmental data and the environmental language. The prompt generation unit 340 notifies the communication control unit 310 of the prompt containing the generated image.

[0069] Figure 12 is the fifth figure showing a specific example of processing by the prompt generation unit. As described above, the prompt generation unit 340 adjusts the image editing parameters based on the environmental data notified from the environmental data acquisition unit 320 (in this case, the item values ​​of items that affect the air quality of the target environment), includes them in the prompt, and notifies the communication control unit 310.

[0070] The example in Figure 12 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the item values ​​of items that affect the air quality of the target environment, such as "season" or "time," as environmental data. As shown in the example in Figure 12, when the item values ​​of items that affect the air quality of the target environment, such as "season" or "time," are notified as environmental data, the prompt generation unit 340 adjusts the image editing parameters "brightness," "contrast," "saturation," and "hue." The prompt generation unit 340 also adjusts the image editing parameters such as "filter," "blur," "sharpness," "shadow," and "noise reduction."

[0071] Figure 13 is the sixth figure showing a specific example of processing by the prompt generation unit. As described above, the prompt generation unit 340 adjusts the image editing parameters based on the environmental data notified from the environmental data acquisition unit 320 (in this case, the item value of an item indicating the air condition of the target environment), includes it in the prompt, and notifies the communication control unit 310.

[0072] The example in Figure 13 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the item values ​​of items indicating the air conditions of the target environment, such as "temperature" or "humidity". As shown in the example in Figure 13, when the environmental data notifies the item values ​​of items indicating the air conditions of the target environment, such as "temperature" or "humidity", the prompt generation unit 340 adjusts the image editing parameters "brightness", "contrast", and "saturation". The prompt generation unit 340 also adjusts the image editing parameters such as "hue", "filter", "blur", "sharpness", "shadow", or "texture", and "noise reduction".

[0073] Figure 14 is the seventh figure showing a specific example of processing by the prompt generation unit. As described above, the prompt generation unit 340 adjusts the image editing parameters based on the environmental data notified from the environmental data acquisition unit 320 (in this case, the item values ​​of items indicating the air quality of the target environment), includes them in the prompt, and notifies the communication control unit 310.

[0074] The example in Figure 14 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the item values ​​of items indicating the air conditions of the target environment, such as "wind speed" or "rainfall." As shown in the example in Figure 14, when the environmental data notifies the item values ​​of items indicating the air conditions of the target environment, such as "wind speed" or "rainfall," the prompt generation unit 340 adjusts the image editing parameters "brightness," "contrast," and "saturation." The prompt generation unit 340 also adjusts image editing parameters such as "hue," "filter," "blur," "sharpness," "effects," "texture," and "noise reduction."

[0075] Figure 15 is the eighth figure showing a specific example of processing by the prompt generation unit. As described above, the prompt generation unit 340 adjusts the image editing parameters based on the environmental data notified from the environmental data acquisition unit 320 (in this case, the item values ​​of items that affect the air quality of the target environment), includes them in the prompt, and notifies the communication control unit 310.

[0076] The example in Figure 15 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the item values ​​of items that affect the air quality of the target environment, such as "location". As shown in the example in Figure 15, when the environmental data notifies the item values ​​of items that affect the air quality of the target environment, such as "location", the prompt generation unit 340 adjusts the image editing parameters "hue", "filter", and "texture". The prompt generation unit 340 also adjusts the image editing parameters "blur", "shadows and highlights", "noise", and "brightness and contrast (or shadows)".

[0077] Figure 16 is the ninth figure showing a specific example of processing by the prompt generation unit. For example, when the prompt generation unit 340 generates text to be included in the prompt, it further identifies a style based on the generated text. Then, the prompt generation unit 340 adjusts the image editing parameters based on the identified "style".

[0078] The example in Figure 16 shows how the prompt generation unit 340, after identifying the painting style as "watercolor style," adjusted image editing parameters such as "blur," "saturation," "texture addition," "brush stroke," "hue," and "transparency adjustment."

[0079] The example in Figure 16 shows how the prompt generation unit 340, after identifying the painting style as "oil painting style," adjusted image editing parameters such as "sharpness," "filter," "brush stroke," "color adjustment," "texture addition," and "shadows and highlights."

[0080] The example in Figure 16 shows how the prompt generation unit 340, after identifying the style as "abstract painting," adjusted image editing parameters such as "saturation," "filter," "hue," "noise addition," "blur and sharpness," and "layer blend mode."

[0081] In this explanation, we have described a case where the prompt generation unit 340 identifies the "art style" based on the text included in the prompt. However, the prompt generation unit 340 may also identify the "art style" based on user specifications.

[0082] <Flowchart of image generation process by the image generation system> The image generation process flow by the image generation system 100 will be explained. Figure 17 is the first flowchart showing the image generation process flow by the image generation system.

[0083] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (codes 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0084] In step S1702, the information processing device 120 generates an environmental language by converting the acquired environmental data into natural language that includes sensory expressions.

[0085] In step S1703, the information processing device 120 generates a prompt based on either or both of the generated environmental language and the acquired environmental data. The prompt generated by the information processing device 120 includes: • Words or sentences that describe images that co-occur from either or both environmental data and / or environmental language. • Images that represent images co-occurring from either or both environmental data and / or environmental language. • Image editing parameters according to environmental data This includes, among others.

[0086] In step S1704, the information processing device 120 sends the generated prompt to the server device 110. As a result, the generation AI 111 of the server device 110 generates an image corresponding to the prompt.

[0087] In step S1705, the information processing device 120 acquires the image generated by the generation AI 111 of the server device 110 and displays it to the user.

[0088] In step S1706, the information processing device 120 determines whether or not to terminate the image generation process. If it is determined in step S1706 to continue the image generation process (if the answer in step S1706 is NO), the process returns to step S1701.

[0089] On the other hand, if it is determined in step S1706 that the image generation process should be terminated (i.e., if the answer in step S1706 is YES), the image generation process is terminated.

[0090] <Example Display> This section describes an example of a display where the information processing device 120 displays an image generated by the server device 110 based on a prompt generated by the information processing device 120. Figure 18 is the first figure showing an example of a display screen.

[0091] As shown in Figure 18, the display screen 1800 has an area 1810 for displaying an image, an area 1820 for displaying environmental data, an area 1830 for displaying image editing parameters, and an area 1840 for displaying text included in the prompt used to generate the image. The display screen 1800 may also have an area for displaying the environmental language instead of, or in addition to, the area 1820 for displaying environmental data.

[0092] <Summary> As is clear from the above description, the information processing apparatus 120 according to the first embodiment is - Obtain environmental data that includes item values ​​for items indicating the air quality of the target environment or item values ​​for items that affect the air quality of the target environment. • Environmental language is generated by converting environmental data into natural language that includes sensory expressions. • Generate prompts containing text, sentences, or images that represent images co-occurring from the environment language or environment data. • Use prompts to run the generation AI and display an image generated by the generation AI that shows the air quality of the target environment.

[0093] As a result, the information processing device 120 according to the first embodiment can generate an image that intuitively represents the state of the air in the target environment.

[0094] [Second Embodiment] In the first embodiment described above, an image that intuitively represents the air quality of the target environment was displayed on the display screen 1800. However, the image displayed on the display screen is not limited to an image that intuitively represents the air quality of the target environment. For example, an image that intuitively represents the opposite air quality of the target environment (the air quality of a comparison environment) may be displayed in comparison. This allows the user to grasp the air quality of the target environment more clearly. The second embodiment will now be described, focusing on the differences from the first embodiment.

[0095] <Functional Configuration of Information Processing Devices> The functional configuration of the information processing device 120 according to the second embodiment will now be described. Figure 19 is a second diagram showing an example of the functional configuration of the information processing device. The difference from the functional configuration described using Figure 3 in the first embodiment is that comparison environment data is input to the environment data acquisition unit 320 from the comparison environment data storage unit 1910.

[0096] The comparison environment data storage unit 1910 stores the comparison environment data. The comparison environment data is The environmental data acquisition unit 320 receives each item included in the environmental data notified by the communication control unit 310, and • Each item included in the environmental data measured by sensors 1 to n (code 130_1 to code 130_n), and • Each item included in the environmental data entered by the user, It has similar items.

[0097] However, the item values ​​for each item in the comparative environment data are • The item values ​​of each item included in the environmental data notified from the communication control unit 310, and, • The item values ​​of each item included in the environmental data measured by sensors 1 to n (code 130_1 to code 130_n), and • The item values ​​of each item included in the environmental data entered by the user. This means that the item values ​​are different (for example, item values ​​in a poor environment). Specifically, the item values ​​for each item in the comparative environment data are item values ​​that indicate the opposite air condition to the air condition of the target environment (air condition of the comparative environment), or item values ​​that influence the opposite air condition to the air condition of the target environment (air condition of the comparative environment).

[0098] The environmental data acquisition unit 320 acquires environmental data, notifies the environmental language generation unit 330 and the prompt generation unit 340 of the acquired environmental data, and reads out comparison environmental data. The environmental data acquisition unit 320 also notifies the environmental language generation unit 330 and the prompt generation unit 340 of the read-out comparison environmental data.

[0099] The prompt generation unit 340 generates prompts for generating an image that intuitively represents the air quality of the target environment, as well as prompts for generating an image that intuitively represents the air quality of the comparison environment. As a result, the output unit 350 can acquire the image that intuitively represents the air quality of the target environment and the image that intuitively represents the air quality of the comparison environment from the server device 110 via the communication control unit 310, and display them in comparison on the display screen.

[0100] <Example Display> This section describes an example of a display when the information processing device 120 displays an image generated by the server device 110 based on a prompt generated by the information processing device 120. Figure 20 is a second diagram showing an example of the display screen.

[0101] As shown in Figure 20, the display screen 2000 has an area 2010 for displaying an image that intuitively represents the air quality of the target environment, and an area 2011 for displaying text included in the prompt used to generate the image. The display screen 2000 also has an area 2020 for displaying an image that intuitively represents the air quality of the comparison environment, and an area 2021 for displaying text included in the prompt used to generate the image.

[0102] By displaying images that intuitively represent the air quality of the target environment and images that intuitively represent the air quality of a comparison environment, users can, for example, more clearly understand that the air quality of the target environment is that of an ideal environment.

[0103] In the example shown in Figure 20, an image representing the air quality of a poor environment was displayed as an image that intuitively represents the air quality of a comparison environment. However, if the target environment is a poor environment, an image representing the air quality of an ideal environment may be displayed. This makes it possible to provide a user interface that, for example, controls the air conditioning to create the ideal environment when the user selects the image representing the air quality of the ideal environment.

[0104] <Summary> As is clear from the above description, the information processing device 120 according to the second embodiment is When displaying an image showing the air quality of the target environment, an image showing the air quality of a comparison environment different from the target environment will be displayed.

[0105] As a result, according to the second embodiment, the user will be able to understand the air quality of the target environment more clearly.

[0106] [Third Embodiment] In the first and second embodiments described above, the case in which the information processing device 120 causes the generation AI 111 to generate a still image was described. However, the information processing device 120 may also cause the generation AI 111 to generate a moving image. The third embodiment will now be described, focusing on the differences from the first embodiment.

[0107] <Functional Configuration of Information Processing Devices> The functional configuration of the information processing device 120 according to the third embodiment will now be described. Figure 21 is a third diagram showing an example of the functional configuration of the information processing device. The difference from the functional configuration described using Figure 3 in the first embodiment is that the functional configuration shown in Figure 21 has an operation control language generation unit 2110.

[0108] The control table storage unit 2120 stores the control table. The control table mainly contains items similar to those included in the environmental data notified to the environmental data acquisition unit 320 from the communication control unit 310. The control table also has an action control language that represents the degree of movement of each component in the video, which is associated with the item value of each item. The components referred to here are the components included in the video generated by the generation AI (for example, clouds, plants, flowers, rivers, etc.).

[0109] The operation control language generation unit 2110 generates operation control language based on the environment data notified by the environment data acquisition unit 320 by referring to the control table stored in the control table storage unit 2120. The operation control language generation unit 2110 notifies the prompt generation unit 340 of the generated operation control language.

[0110] The prompt generation unit 340 adds the operation control language notified by the operation control language generation unit 2110 to the words or sentences included in the prompt.

[0111] <Specific example of processing by the operation control language generation unit> A specific example of processing by the operation control language generation unit 2110, one of the parts of the information processing device 120 according to the third embodiment, will be described. Figure 22 is a diagram showing a specific example of processing by the operation control language generation unit. As shown in Figure 22, the control table storage unit 2120 stores, for example, control tables 2210 to 2230.

[0112] Control table 2210 is an example of a table that defines the control language for operation when the component is "cloud". In the example of control table 2210, the control language is defined for each item value of the item "wind speed" and for each altitude.

[0113] Control table 2220 is an example of a table that defines the control language for when the component is "grass, flowers". In the example of control table 2220, the control language is defined for each item value of the item "wind speed at ground level".

[0114] Control table 2230 is an example of a table that defines the control language for operation when the component is "river". In the example of control table 2230, the control language is defined for each item value of the item "rainfall".

[0115] For example, if the operation control language generation unit 2110 receives environmental data such as wind speed at a specific altitude from the environmental data acquisition unit 320, it refers to the control table 2210. The operation control language generation unit 2110 generates operation control language that represents the degree of cloud movement (flow) according to the wind speed at the specific altitude and notifies the prompt generation unit 340. This allows the prompt generation unit 340 to add the operation control language to the element "cloud" in the sentence included in the prompt.

[0116] The motion control language generation unit 2110, for example, when the wind speed at ground level is notified as environmental data from the environmental data acquisition unit 320, refers to the control table 2220. The motion control language generation unit 2110 generates motion control language that represents the degree of movement (swaying) of plants and flowers according to the wind speed at ground level, and notifies the prompt generation unit 340. As a result, the prompt generation unit 340 can add the motion control language to the elements of the text included in the prompt, namely "plants and flowers".

[0117] The operation control language generation unit 2110, for example, when it receives rainfall as environmental data from the environmental data acquisition unit 320, refers to the control table 2220. The operation control language generation unit 2110 generates operation control language that represents the degree of river operation (flow) according to the rainfall and notifies the prompt generation unit 340. As a result, the prompt generation unit 340 can add operation control language if the sentence in the prompt contains the element "river".

[0118] <Examples of prompts and videos> An example of a prompt generated by the information processing device 120 according to the third embodiment, and an example of a video generated by the generation AI 111 of the server device 110 based on the prompt will be described. Figure 23 shows an example of a prompt (text) and an example of a video.

[0119] Of these, code 2310 shows that the generated prompt includes "clouds" as a component and "quickly" as the operation control language. Code 2311 shows an example of a video (a video showing clouds moving quickly) generated by the generation AI 111 of the server device 110 based on the prompt containing the text shown in code 2310.

[0120] Code 2320 indicates that the generated prompt includes "river," "flower," and "cloud" as components, but no operation control language is attached to any of the components. In this case, the generation AI 111 of the server device 110 generates a video in which each component operates based on the default operation control language. Code 2321 shows an example of a video generated by the server device 110 based on the prompt containing the text shown in code 2320 (a video showing a river and clouds flowing at their respective default speeds, and a flower swaying at its default speed).

[0121] Code 2330 shows that the generated prompt includes "river" as a component and "strongly" as the operation control language. Code 2331 shows an example of a video (a video depicting a river flowing vigorously) generated by the generation AI 111 of the server device 110 based on the prompt containing the text shown in code 2330.

[0122] <Flowchart of image generation process by the image generation system> The image generation process flow of the image generation system 100 equipped with the information processing device 120 according to the third embodiment will be described. Figure 24 is a second flowchart showing the image generation process flow of the image generation system.

[0123] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (codes 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0124] In step S1702, the information processing device 120 generates an environmental language by converting the acquired environmental data into natural language that includes sensory expressions.

[0125] In step S2401, the information processing device 120 generates an operation control language based on the acquired environmental data.

[0126] In step S1703, the information processing device 120 generates a prompt based on either or both of the generated environment language and the acquired environment data. The information processing device 120 adds the corresponding operation control language to the sentence components included in the generated prompt. The prompt generated by the information processing device 120 includes: • Words or sentences that represent images co-occurring from either or both environmental data and / or environmental language (with behavior control language attached to the components), • Images that represent images co-occurring from either or both environmental data and / or environmental language. • Image editing parameters according to environmental data This includes, among others.

[0127] In step S2403, the information processing device 120 sends the generated prompt to the server device 110. As a result, the generation AI 111 of the server device 110 generates an image (moving image) corresponding to the prompt.

[0128] In step S2404, the information processing device 120 acquires the image (moving image) generated by the generation AI 111 of the server device 110 and displays it to the user.

[0129] In step S1706, the information processing device 120 determines whether or not to terminate the image generation process. If it is determined in step S1706 to continue the image generation process (if the answer in step S1706 is NO), the process returns to step S1701.

[0130] On the other hand, if it is determined in step S1706 that the image generation process should be terminated (i.e., if the answer in step S1706 is YES), the image generation process is terminated.

[0131] <Summary> As is clear from the above description, the information processing device 120 according to the third embodiment performs the processing performed by the information processing device 120 according to the first embodiment, Based on environmental data, generate an operation control language that describes the degree of operation of the components. • Add action control language to the elements of the text included in the prompt. • Use prompts to operate the generating AI and display a video showing the air quality of the target environment generated by the generating AI, in which the components operate based on the operation control language.

[0132] As a result, according to the information processing device 120 of the third embodiment, it becomes possible to generate a moving image that intuitively represents the air quality of the target environment.

[0133] [Fourth Embodiment] In each of the above embodiments, images were acquired by operating the generation AI 111 on the server device 110. In contrast, in this embodiment and subsequent embodiments, images are acquired by performing additional training on the generation AI 111 on the server device 110 during the learning phase and then operating it during the generation phase. The fourth embodiment will now be described, focusing on the differences from the above embodiments.

[0134] <System configuration of the image generation system> The system configuration in the learning phase of an image generation system to which the information processing device according to the fourth embodiment is applied will be described. Figure 25 is a second diagram showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the learning phase.

[0135] As shown in Figure 25, the image generation system 2500 comprises a server device 110, an information processing device 120, and a server device 2540. In the image generation system 2500, the information processing device 120, the server device 110, and the server device 2540 are connected to each other via a network 150.

[0136] In the learning phase, the server device 110 has a generating AI 111 and a fine-tuning unit 2511. When the server device 110 receives training data from the information processing device 120 via the network 150, it uses the fine-tuning unit 2511 to perform additional training on the generating AI 111. As a result, the fine-tuning unit 2511 generates a tuned model corresponding to each piece of training data.

[0137] In the learning phase, the information processing device 120 generates training data and instructs the generating AI 111 to perform additional training. Specifically, the information processing device 120 acquires a group of images to be included in the training data (an example of a specific group of images) from the server device 2540. The information processing device 120 generates training data that includes the acquired group of images and captions indicating each component contained in each image. The information processing device 120 transmits the generated training data to the server device 110. The information processing device 120 instructs the generating AI 111 to perform additional training using a model (LoRA in this embodiment) for generating each image (an example of each specific image) included in the generated training data.

[0138] During the learning phase, the information processing device 120 calculates a score for each component in the training data based on a score indicating the importance of each component within the image, and stores it in association with the training data.

[0139] The server device 2540 functions as an information provision unit 2541 and provides images to be included in the training data to the information processing device 120 via the network 150.

[0140] <Functional Configuration of Information Processing Devices> The functional configuration of the information processing device 120 during the learning phase will be described. Figure 26 is the fourth figure showing an example of the functional configuration of the information processing device. An information processing program is installed in the information processing device 120, and when this information processing program is executed during the learning phase, the information processing device 120 performs the following actions: • Communication control unit 310, • Image data collection unit 2610, • Caption generation unit 2620, • Training data generation unit 2630, • Additional learning section 2640, It functions as such.

[0141] The communication control unit 310 acquires images from the server device 2540 via the network 150 and notifies the image data collection unit 2610. The communication control unit 310 acquires training data and instructions for additional training from the additional training unit 2640 and transmits them to the server device 110 via the network 150.

[0142] The image data acquisition unit 2610 stores the images notified by the communication control unit 310 in the image data storage unit 2650.

[0143] The caption generation unit 2620 reads images stored in the image data storage unit 2650 and performs image recognition processing on the read images to recognize each component contained in the read images and calculate the confidence level of each recognized component. As a result, the caption generation unit 2620 generates captions that indicate each component contained in each image. The caption generation unit 2620 notifies the training data generation unit 2630 of the read images, the generated captions, and the calculated confidence levels of each component.

[0144] The training data generation unit 2630 obtains the image, caption, and confidence level of each component from the caption generation unit 2620. The training data generation unit 2630 classifies the image notified by the caption generation unit 2620 into multiple groups according to the type of image content. Based on the confidence level of each component corresponding to each image belonging to the classified group, the training data generation unit 2630 calculates a component-specific score for each group.

[0145] The training data generation unit 2630 generates training data that includes images belonging to the classified group and the captions corresponding to those images. The training data generation unit 2630 associates the generated training data with component-specific scores calculated for that group and stores it in the training data storage unit 2660.

[0146] The additional learning unit 2640 sequentially reads the training data generated for each group and notifies the communication control unit 310 of the instruction for additional learning. As a result, the communication control unit 310 can sequentially transmit the read training data to the server device 110 and instruct it to perform additional learning on the generated AI 111 using the generated training data.

[0147] <Specific examples of processing performed by each part of the information processing device> Specific examples of processing performed by each part of the information processing device 120 (here, the caption generation unit 2620 and the training data generation unit 2630) will be described.

[0148] (1) Specific example of processing by the caption generation unit Figure 27 shows a specific example of processing by the caption generation unit. As described above, the caption generation unit 2620 reads the image stored in the image data storage unit 2650, and performs image recognition processing on the read image to recognize each component contained in the read image and calculate the confidence level of each recognized component.

[0149] In Figure 27, reference numeral 2710 indicates an image read from the image data storage unit 2650 by the caption generation unit 2620. In Figure 27, reference numeral 2720 indicates that the caption generation unit 2620 performs image recognition processing on the image read from the caption generation unit 2620. ·"sunsetcloud", ·"scenery", ·"sun", ·"sky", · "cloud", ·"mountain", This shows how each component is recognized. Also, in Figure 27, reference numeral 2720 indicates how the caption generation unit 2620 calculates the confidence level of each component when it performs image recognition processing on the read image. The caption generation unit 2620 notifies the training data generation unit 2630 of the read image, the captions indicating each component contained in the image, and the calculated confidence levels of each component.

[0150] (2) Specific example of processing by the training data generation unit Figure 28 shows a specific example of processing by the training data generation unit. As described above, the training data generation unit 2630 obtains images, captions indicating each component contained in the images, and the confidence level of each component from the caption generation unit 2620, and classifies the obtained images into multiple groups according to the type of image content. Based on the confidence level of each component corresponding to each image belonging to the classified group, the training data generation unit 2630 calculates a score for each component for each group.

[0151] In Figure 28, reference numeral 2710 indicates the image notified by the caption generation unit 2620, and reference numeral 2720 indicates the caption notified by the caption generation unit 2620. Reference numeral 2730 indicates the confidence level of each component notified by the caption generation unit 2620.

[0152] In Figure 28, reference numeral 2800 indicates that the learning data generation unit 2630 is, From the image shown in reference numeral 2710, images with the image content type "sunset" are extracted and classified into the group "sunset" (see reference numeral 2810), • Assign the caption corresponding to the image content type "Sunset" to the image content type "Sunset" (see reference numeral 2820). This shows how training data (for sunsets) was generated.

[0153] In Figure 28, reference numeral 2830 denotes the learning data generation unit 2630, From the confidence levels of each component shown in symbol 2730, read out the confidence level of each component corresponding to the image belonging to the group of image content types "sunset". • The confidence scores for each component are summed up and divided by the number of images belonging to the group "Sunset" in terms of image content type. This shows how component-specific scores are calculated and mapped to the training data (for sunsets). Note that when mapping component-specific scores to the training data (for sunsets), components with scores below a predetermined threshold may be excluded.

[0154] <Server device details> Details of the generation AI 111 and the fine-tuning unit 2511 of the server device 110 in the fourth embodiment will be described. Figure 29 is a diagram showing the details of the generation AI and the fine-tuning unit.

[0155] As shown in Figure 29, the generated AI 111 includes a VAE (Variational Auto-Encoder) 2911, an encoder 2912, a decoder 2913, and a VAE 2914. The fine-tuning unit 2511 includes a LoRA (Low-Rank Adaptation) 2920. The LoRA 2920 is, for example, • The LoRA2921 for sunsets used when performing additional training on the generated AI111 using training data (for sunsets), • The LoRA2922 for mackerel clouds is used when performing additional training on the generating AI111 using the training data (for mackerel clouds), • LoRA2923 for landscape images is used when performing additional training on the generating AI using training data (for landscape images), It has.

[0156] For example, when performing additional training on a generative AI using LoRA2921 for sunsets, The image shown at code 2810 and the caption shown at code 2820 of the training data (for sunsets) shown at code 2800 in Figure 28 are input to the generating AI 111. The model parameters of the Generator AI111 are fine-tuned so that the image reproduced by the Generator AI111 matches the image shown in reference numeral 2810 (the image input to the Generator AI111).

[0157] <Flow of additional learning processing by the image generation system> This section describes the flow of additional learning processing during the learning phase using the image generation system 2500. Figure 30 is a flowchart showing the flow of additional learning processing by the image generation system.

[0158] In step S3001, the information processing device 120 collects images provided by the server device 2540.

[0159] In step S3002, the information processing device 120 performs image recognition processing on the collected image to recognize each component contained in the image and calculates the confidence level of each recognized component. The information processing device 120 generates a caption indicating each component contained in the image.

[0160] In step S3003, the information processing device 120 generates groups of additional learning units by classifying the collected images into multiple groups according to the type of image content.

[0161] In step S3004, the information processing device 120 calculates a component-specific score for each group based on the reliability of each component.

[0162] In step S3005, the information processing device 120 generates training data, which associates images with captions, for each group of additional training units.

[0163] In step S3006, the information processing device 120 sequentially transmits the training data generated for each group of additional training units to the server device 110, thereby performing additional training on the generated AI 111 using each LoRA of the fine-tuning unit 2511.

[0164] In step S3007, the information processing device 120 determines whether or not to terminate the additional learning process. If it is determined in step S3007 to continue the additional learning process (i.e., if the answer in step S3007 is NO), the process returns to step S3001.

[0165] On the other hand, if it is determined in step S3007 to terminate the additional learning process (i.e., if the answer in step S3007 is YES), the additional learning process is terminated.

[0166] <System configuration of the image generation system> The system configuration in the generation phase of an image generation system to which the information processing device according to the fourth embodiment is applied will be described. Figure 31 is a third figure showing an example of the system configuration of an image generation system, and is a diagram showing the system configuration in the generation phase.

[0167] As shown in Figure 31, the image generation system 3100 comprises a server device 110, an information processing device 120, and a server device 140. In the image generation system 3100, the information processing device 120 and the server devices 110 and 140 are communicated with each other via a network 150.

[0168] In the generation phase, the server device 110 has a generation AI 111 and a tuned model 3110. In the generation phase, when the server device 110 receives a prompt from the information processing device 120 via the network 150, it operates the generation AI 111 using the tuned model 3110 to generate an image corresponding to the prompt. The server device 110 transmits the generated image to the information processing device 120 via the network 150.

[0169] The information processing device 120 is the same as the information processing device 120 described with reference to Figure 1 in the first embodiment above, so its description is omitted here. However, in the case of the information processing device 120 according to the fourth embodiment, the prompt to be sent to the server device 110 includes: • Specify the tuned model 3110 (running LoRA) to be used, • When using a tuned model, the weights (each weight corresponding to the type of image content in each of the multiple training datasets (weights for each execution of LoRA)) and It shall include the following.

[0170] Since the server device 140 is the same as the server device 140 described with reference to Figure 1 in the first embodiment described above, its description will be omitted here.

[0171] <Functional Configuration of Information Processing Devices> The functional configuration of the information processing device 120 in the generation phase according to the fourth embodiment will now be described. Figure 32 is a fifth figure showing an example of the functional configuration of the information processing device. The difference from the functional configuration described using Figure 3 in the first embodiment is that it has a weight calculation unit 3210.

[0172] The weight calculation unit 3210 obtains the words or sentences contained in the prompts generated by the prompt generation unit 340 and extracts the components contained in each training data. The weight calculation unit 3210 reads the component-specific scores associated with the training data stored in the training data storage unit 2660 and extracts the components corresponding to the read component-specific scores from the prompts.

[0173] The weight calculation unit 3210 calculates the sum of the scores for each component in each training data set (total score) from the scores of each training data set corresponding to the components extracted from the prompt.

[0174] The weight calculation unit 3210 compares the total scores of each training data and selects the training data with the higher total score to designate the corresponding LoRA as the execution LoRA. The weight calculation unit 3210 calculates the weight for each execution LoRA based on the total score of each selected training data.

[0175] The weight calculation unit 3210 notifies the prompt generation unit 340 of the weight of each execution LoRA. This allows the prompt generation unit 340 to generate a prompt that includes the specification of the execution LoRA and the weight of each execution LoRA.

[0176] As a result, according to the information processing device 120 of the fourth embodiment, based on the generated prompt, it is possible to select a generator AI that has been pre-trained using each of the multiple training data and operate it based on the assigned weights. Here, "selecting a generator AI that has been pre-trained" refers to, for example, selecting a combination of execution LoRAs from a generator AI that includes multiple LoRAs generated by pre-training.

[0177] <Specific example of processing by the weight calculation unit> A specific example of processing by the weight calculation unit 3210 will be described. Figure 33 is the first diagram showing a specific example of processing by the weight calculation unit.

[0178] In Figure 33, reference numeral 3310 denotes an example of text included in a prompt generated by the prompt generation unit 340.

[0179] The weight calculation unit 3210 acquires the text (code 3310) contained in the prompt and extracts the components corresponding to the component-specific scores (codes 3331 to 3333) associated with each training data from the acquired text. In the text (code 3310) contained in the prompt, the underlined components indicate the components extracted by the weight calculation unit 3210 as components corresponding to the component-specific scores (codes 3331 to 3333) associated with each training data.

[0180] The weight calculation unit 3210 calculates the sum of the scores of each component for each training data set (total score) from the scores of each training data corresponding to the components extracted from the prompt (underlined in reference numeral 3310). In Figure 33, reference numeral 3321 is, • The total score "399" was calculated from the scores corresponding to the components extracted from the prompt (underlined in code 3310) among the component-specific scores (code 3331) associated with the training data (for mackerel clouds). • The total score "376" was calculated from the scores corresponding to the components extracted from the prompt (underlined in code 3310) among the component-specific scores (code 3332) associated with the training data (for sunsets). • The total score "347" was calculated from the scores corresponding to the components extracted from the prompt (underlined in code 3310) among the component-specific scores (code 3333) associated with the training data (for landscape paintings). This indicates that.

[0181] The weight calculation unit 3210 compares the total scores of each training data and selects the training data with the higher total score, thereby designating the corresponding LoRA as the running LoRA. In the example in Figure 33, for the sake of space, the total scores of three training data are calculated, and these three training data are selected, resulting in the designation of the LoRA for mackerel clouds, the LoRA for sunsets, and the LoRA for landscape paintings as the running LoRAs (see reference numeral 3322).

[0182] The weight calculation unit 3210 calculates the weight for each LoRA based on the total score of each selected training data. In Figure 33, code 3323 calculates the weight of each execution LoRA corresponding to each training data from the ratio of the total scores of the training data (sardine clouds), training data (sunset), and training data (landscape painting). 0.34, 0.36, 0.30, This shows how the calculation was performed.

[0183] The weight calculation unit 3210 notifies the prompt generation unit 340 of the weight of each executed LoRA. Reference numeral 3324 denotes: • LoRA weights for mackerel clouds = 0.34 • LoRA weight for sunset = 0.36 • LoRA weight for landscape painting = 0.30 This shows how the prompt generation unit 340 was notified.

[0184] <Flowchart of image generation process by the image generation system> The image generation process flow in the generation phase by the image generation system 3100 will be explained. Figure 34 is a third flowchart showing the image generation process flow by the image generation system. Note that the difference from the first flowchart shown in Figure 17 is step S3401.

[0185] In step S3401, the information processing device 120 extracts components from the text contained in the generated prompt that correspond to the component-specific scores associated with each training data. The information processing device 120 calculates the sum of the scores for each component for each training data (total score) from the scores corresponding to the extracted components. The information processing device 120 specifies the execution LoRA based on the total score for each training data and calculates the weight of each execution LoRA. The information processing device 120 includes the specification of the execution LoRA and the weight of each execution LoRA in the prompt.

[0186] <Summary> As is clear from the above explanation, the information processing device 120 according to the fourth embodiment, in the learning phase, • Recognize each component contained in the collected image and calculate the confidence level of each recognized component. The collected images are classified into multiple groups according to the type of image content, and component-specific scores are calculated for the components contained in the images belonging to each group. • For each group, training data is generated in which images and captions describing each component contained in the images are associated, and component-specific scores are associated with each piece of training data. • Using multiple generated training datasets, a tuned model is created by performing additional training on the generative AI using the corresponding LoRA.

[0187] Furthermore, in the generation phase, the information processing apparatus 120 according to the fourth embodiment performs the following: • Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. • Generate prompts containing text, sentences, or images that represent images co-occurring from the environment language or environment data. Based on the text elements contained in the generated prompt, specify the execution LoRA and calculate the weight for each execution LoRA. The system uses prompts that specify the execution LoRA and the weights of each execution LoRA to run the generating AI. Furthermore, by running the generating AI, the system acquires an image representing the air quality of the target environment, which is then displayed to the user.

[0188] In this way, by operating a generating AI that has undergone additional training using multiple training data generated for each type of image content, the information processing device 120 according to the fourth embodiment can generate images that intuitively represent the state of the air with high accuracy.

[0189] [Fifth Embodiment] In the fourth embodiment described above, when calculating the weights of the execution LoRA, the component-specific scores associated with the training data were referenced. However, the method for calculating the weights of the execution LoRA is not limited to this, and the weights of the execution LoRA may be calculated using other methods. The fifth embodiment will now be described, focusing on the differences from the fourth embodiment.

[0190] <Specific example of processing by the weight calculation unit> A specific example of processing by the weight calculation unit 3210 of the information processing device 120 according to the fifth embodiment will be described. Figure 35 is a second figure showing a specific example of processing by the weight calculation unit. The difference from the first figure shown in Figure 33 is the reference numerals 3421 and 3422.

[0191] As shown by reference numeral 3421, in the information processing apparatus 120 according to the fifth embodiment, the weight calculation unit 3210 uses an LLM (Large Language Model) to determine the importance of the words represented by the execution LoRA in the sentences included in the prompt. Note that LLM is an abbreviation for Large Language Model and refers to a large-scale language model.

[0192] The example in FIG. 35 shows the case where sardine cloud LoRA, sunset LoRA, and landscape painting LoRA are specified as the execution LoRA. Further, the example in FIG. 35 shows the case where the importance of the words "sardine cloud", "sunset", and "landscape painting", which are the words represented by the specified execution LoRA, in the sentence (reference numeral 3310) included in the prompt is determined.<第5の実施形態に係る情報処理装置120において、重み算出部3210は、各実行LoRAの重みをプロンプト生成部340に通知する。符号3422は、>

[0193] According to the LLM, since "sunset" is "an element that determines the overall atmosphere and affects light and color with respect to other elements", the importance is determined to be "50%". According to the LLM, since "sardine cloud" is "beautifully illuminated by the light of the sunset and its shadow gives depth to the landscape", the importance is determined to be "30%". According to the LLM, since "landscape painting" is "enhanced by the sunset and sardine cloud and forms the overall beauty, but does not play as central a role as the sunset and sardine cloud", the importance is determined to be "20%".

[0194]

[0195]

[0196] <まとめ> As is clear from the above description, the information processing apparatus 120 according to the fifth embodiment performs the generation phase, • Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. • Generate prompts containing text, sentences, or images that represent images co-occurring from the environment language or environment data. • Specify the LoRA to execute based on the text elements contained in the generated prompt. The weight of each execution LoRA is calculated by determining the importance of the word represented by each execution LoRA in the sentence contained in the prompt using LLM. By using prompts that include the specified execution LoRA and the weights of each execution LoRA, the generating AI is activated, and an image representing the air quality of the target environment, generated by the generating AI, is displayed.

[0197] In this way, by operating a generating AI that has undergone additional training using training data generated for each type of image content, the information processing device 120 according to the fifth embodiment can generate images that intuitively represent the state of the air with high accuracy, similar to the fourth embodiment.

[0198] [Sixth Embodiment] In the fourth and fifth embodiments described above, the execution LoRA was specified based on the text contained in the generated prompt. However, the method of specifying the execution LoRA is not limited to this, and for example, conditions for specifying the execution LoRA (referred to as environmental conditions) may be set, and the system may be configured to specify the execution LoRA when these environmental conditions are met. The sixth embodiment will now be described, focusing on the differences from the fourth and fifth embodiments described above.

[0199] <Functional Configuration of Information Processing Devices> The functional configuration of the information processing device 120 in the generation phase according to the sixth embodiment will now be described. Figure 36 is the sixth figure showing an example of the functional configuration of the information processing device. The difference from the functional configuration described using Figure 32 in the fourth embodiment is that the weight calculation unit 3210 is replaced with a selection unit 3610.

[0200] The selection unit 3610 obtains the words or sentences contained in the prompts generated by the prompt generation unit 340 and extracts the components contained in each training data. The selection unit 3610 reads the component-specific scores associated with the training data stored in the training data storage unit 2660 and extracts the components corresponding to the read component-specific scores from the prompts.

[0201] The selection unit 3610 calculates the sum of the scores for each component in each training data set (total score) from the scores of each training data set corresponding to the components extracted from the prompt.

[0202] The selection unit 3610 compares the total scores of each training data and selects the training data with the higher total score, thereby designating the corresponding LoRA as the LoRA to be executed. At this time, the selection unit 3610 determines whether or not environmental conditions are attached to the selected training data, and if environmental conditions are attached, it determines whether or not the text included in the generated prompt is consistent with the environmental conditions.

[0203] The selection unit 3610 determines that environmental conditions are attached to the selected training data, and if it determines that the text included in the generated prompt is consistent with the environmental conditions, it does not change the specified execution LoRA.

[0204] On the other hand, if the selected training data is determined to have environmental conditions attached, and the generated prompt contains text that does not match the environmental conditions, the selection unit 3610 re-specifies the LoRA corresponding to other training data as the execution LoRA.

[0205] Alternatively, if the selection unit 3610 determines that no environmental conditions are attached to the selected training data, it re-designates a LoRA corresponding to any other training data as the execution LoRA.

[0206] The selection unit 3610 evenly distributes weights to each specified execution LoRA, or each re-specified execution LoRA, and notifies the prompt generation unit 340 of the weights evenly distributed to each execution LoRA.

[0207] <Specific examples of processing by the selection unit> A specific example of processing by the selection unit 3610 will be explained. Figure 37 is a diagram showing a specific example of processing by the selection unit.

[0208] In Figure 37, reference numeral 3310 is the same as reference numeral 3310 shown in Figure 33, and reference numerals 3321 and 3322 are the same as reference numerals 3321 and 3322 shown in Figure 33, so their explanation is omitted here.

[0209] As shown in reference numeral 3721, the selection unit 3610 determines whether or not environmental conditions are attached to the selected training data. Here, the selection unit 3610 selects training data (for mackerel clouds), training data (for sunsets), and training data (for landscape paintings). Also, as shown in reference numerals 3731 to 3733, • The training data (for mackerel clouds) does not have environmental conditions attached. • The training data (for sunsets) includes the environmental conditions: time = evening, weather = sunny. • The training data (for landscape painting) has the environmental condition "location = anywhere" assigned to it. Let's assume that.

[0210] In this case, the selection unit 3610 is • The training data (for sunsets) has environmental conditions attached, and if it is determined that the text included in the prompt satisfies the attached environmental conditions, the specified result of designating the corresponding sunset LoRA as the LoRA to be executed will not be changed. · The learning data (for landscape paintings) has environmental conditions assigned to it, and it is determined that the text included in the prompt satisfies the assigned environmental conditions, and the specified result that designates the corresponding LoRA for landscape paintings as the execution LoRA is not changed. · Since the learning data (for sardine clouds) does not have environmental conditions assigned to it, the LoRA for temperature, which is any other LoRA, is reassigned as the execution LoRA (see reference numeral 3721).

[0211] As shown by reference numeral 3722, the selection unit 3610 evenly distributes the weights to each execution LoRA (here, the LoRA for sunset, the LoRA for landscape paintings, and the LoRA for temperature) (for example, evenly distributes them by 0.3 each).

[0212] As shown by reference numeral 3723, the selection unit 3610 uses, as the weights of each specified execution LoRA or each re-specified execution LoRA (an example of a predetermined weight), · Weight of the LoRA for sunset = 0.3, · Weight of the LoRA for landscape paintings = 0.3, · Weight of the LoRA for temperature = 0.3, to notify the prompt generation unit 340.

[0213] <Summary> As is clear from the above description, the information processing apparatus 120 according to the sixth embodiment, in the generation phase, · Acquires environmental data of the target environment and converts it into natural language including sensory expressions to generate environmental language. · Generates a prompt including text or a sentence or an image that represents an image co-occurring with the environmental language or environmental data. · Designates an execution LoRA based on the components of the text included in the generated prompt. · Determines whether the text included in the generated prompt satisfies the environmental conditions assigned to the learning data, and if it is determined that it does not satisfy, re-designates the execution LoRA. · Evenly distributes the weights to each execution LoRA. The system uses prompts that specify the execution LoRA and the weights of each execution LoRA to run the generating AI. Furthermore, by running the generating AI, the system acquires an image representing the air quality of the target environment, which is then displayed to the user.

[0214] Thus, the information processing device 120 according to the sixth embodiment operates the generating AI, which has undergone additional training using training data generated for each type of image content, based on predetermined weights, taking into account the environmental conditions attached to the training data. As a result, the information processing device 120 according to the sixth embodiment can appropriately generate images that intuitively represent the state of the air.

[0215] [Seventh Embodiment] In the fourth embodiment described above, the weight calculation unit 3210 is configured to calculate the weight of each execution LoRA based on the total score. In contrast, in the seventh embodiment, multiple combinations of weights for each execution LoRA are prepared, and the generation AI is operated using each execution LoRA under each combination, and the user evaluates each generated image. This makes it possible to generate images under an appropriate weight combination. The seventh embodiment will now be described, focusing on the differences from the fourth embodiment described above.

[0216] <Functional Configuration of Information Processing Devices> The functional configuration of the information processing device 120 in the generation phase according to the seventh embodiment will now be described. Figure 38 is the seventh figure showing an example of the functional configuration of the information processing device. The differences from the functional configuration described using Figure 32 in the fourth embodiment are that the function of the weight calculation unit 3810 is different from the function of the weight calculation unit 3210 in Figure 32, and that the information processing device 120 according to the seventh embodiment has a weight storage unit 3820.

[0217] The weight calculation unit 3810 obtains the words or sentences contained in the prompts generated by the prompt generation unit 340 and extracts the components contained in each training data. The weight calculation unit 3810 reads the component-specific scores associated with the training data stored in the training data storage unit 2660 and extracts the components corresponding to the read component-specific scores from the prompts.

[0218] The weight calculation unit 3810 calculates the sum of the scores for each component in each training data set (total score) from the scores of each training data set corresponding to the components extracted from the prompt.

[0219] The weight calculation unit 3810 compares the total scores of each training data and selects the training data with the higher total score, thereby designating the corresponding LoRA as the execution LoRA. The weight calculation unit 3810 notifies the prompt generation unit 340 of multiple possible combinations of weights for each designated execution LoRA.

[0220] This allows the Generator AI111 to operate using each execution LoRA under multiple weight combinations. The image corresponding to each combination generated by the Generator AI111 can then be displayed to the user.

[0221] As a result, the user evaluates each image, and the weight calculation unit 3810 receives the evaluation results from the user. The weight calculation unit 3810 stores the combination of weights that the user evaluated most highly among the received evaluation results in the weight storage unit 3820.

[0222] <Specific example of processing by the weight calculation unit> A specific example of processing by the weight calculation unit 3810 of the information processing device 120 according to the seventh embodiment will be described. Figure 39 is a third figure showing a specific example of processing by the weight calculation unit. The differences from the first figure shown in Figure 33 are reference numerals 3921, 3922, 3940, weight calculation unit 3810, and weight storage unit 3820.

[0223] As shown in reference numeral 3921, in the information processing device 120 according to the seventh embodiment, the weight calculation unit 3810 notifies the prompt generation unit 340 of multiple combinations of weights of a specified number of execution LoRAs. In response to the notification of multiple combinations to the prompt generation unit 340, the corresponding images are displayed to the user, and the user inputs their evaluation results. As a result, as shown in reference numeral 3922, the weight calculation unit 3810 stores the weight combination with the highest evaluation among the received evaluation results in the weight storage unit 3820.

[0224] In this configuration, for each specified combination of execution LoRAs, the combination of weights with the highest evaluation is stored in the weight storage unit 3820, making it possible to use that combination of execution LoRAs with appropriate weights thereafter.

[0225] In the example shown in Figure 39, the weight calculation unit 3810 is configured to optimize the weight combination by preparing multiple predetermined weight combinations and storing the weight combination that is most highly rated by the user. However, the method of optimizing the weight combination is not limited to this. For example, each time the weight combination is changed, the evaluation result for the image based on the weight combination before the change is compared with the evaluation result for the image based on the weight combination after the change, and the weight combination is changed in a direction that improves the evaluation result to optimize it.

[0226] <Flowchart of image generation process by the image generation system> The image generation process flow in the generation phase by the image generation system 3100 will be explained. Figure 40 is the fourth flowchart showing the image generation process flow by the image generation system.

[0227] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (codes 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0228] In step S1702, the information processing device 120 generates an environmental language by converting the acquired environmental data into natural language that includes sensory expressions.

[0229] In step S1703, the information processing device 120 generates a prompt based on either the generated environmental language or the acquired environmental data, or both.

[0230] In step S4001, the information processing device 120 extracts components from the text contained in the generated prompt that correspond to the component-specific scores associated with each training data. The information processing device 120 then calculates the sum of the scores of each component for each training data (total score) from the scores corresponding to the extracted components, and specifies the execution LoRA.

[0231] In step S4002, the information processing device 120 determines whether the specified combination of weights for the execution LoRA is stored in the weight storage unit 3820. If it is determined in step S4002 that it is stored (if the answer is YES in step S4002), the process proceeds to step S4003.

[0232] In step S4003, the information processing device 120 sends a prompt to the server device 110 including the specification of the execution LoRA and a combination of weights for the specified execution LoRA, and operates the generation AI 111 using the execution LoRA with that weight combination. As a result, the generation AI 111 of the server device 110 generates an image corresponding to that weight combination.

[0233] In step S1705, the information processing device 120 acquires the image generated by the generation AI 111 and displays it to the user.

[0234] On the other hand, if it is determined in step S4002 that the data is not stored (i.e., the answer in step S4002 is NO), the process proceeds to step S4004.

[0235] In step S4004, the information processing device 120 obtains multiple combinations of weights for the specified execution LoRA.

[0236] In step S4005, the information processing device 120 sequentially sends multiple combinations of the specified execution LoRA weights to the server device 110, including them in the prompt, and sequentially operates the generation AI using the execution LoRA for each combination. As a result, the generation AI of the server device 110 generates an image corresponding to each combination.

[0237] In step S4006, the information processing device 120 acquires images corresponding to each combination generated by the generation AI 111 and displays them to the user. The user then evaluates each image.

[0238] In step S4007, the information processing device 120 stores the combination of weights corresponding to the image with the highest evaluation result input by the user in the weight storage unit 3820. The information processing device 120 also displays the image with the highest evaluation result input by the user on the display screen.

[0239] In step S1706, the information processing device 120 determines whether or not to terminate the image generation process. If it is determined in step S1706 to continue the image generation process (i.e., the answer in step S1706 is NO), the process returns to step S1701.

[0240] On the other hand, if it is determined in step S1706 to terminate the image generation process (i.e., if the answer in step S1706 is YES), the image generation process is terminated.

[0241] <Summary> As is clear from the above description, the information processing apparatus 120 according to the seventh embodiment is • Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. • Generate prompts containing text, sentences, or images that represent images co-occurring from the environment language or environment data. • Specify the LoRA to execute based on the text elements contained in the generated prompt. The generator AI is run by sequentially including multiple combinations of the specified execution LoRA weights in the prompt. The user evaluates multiple images generated by the generation AI, and the combination of weights corresponding to the image with the highest evaluation is stored in the weight storage unit.

[0242] Thus, in the information processing device 120 according to the seventh embodiment, when operating the generating AI, which has undergone additional training using training data generated for each type of image content, the combination of weights highly rated by the user is used. As a result, the information processing device 120 according to the seventh embodiment can generate images that intuitively represent the state of the air with higher accuracy.

[0243] [Eighth Embodiment] In the fourth to seventh embodiments described above, the tuned model 3110 of the server device 110 was described as having multiple arbitrary LoRAs. In contrast, in the eighth embodiment, the multiple LoRAs are categorized into configuration LoRAs, element LoRAs, and environment LoRAs, and the generated AI is operated sequentially using the LoRAs of each category. The eighth embodiment will now be described, focusing on the differences from the fourth to seventh embodiments.

[0244] <System configuration of the image generation system> The system configuration in the learning phase of an image generation system to which the information processing device according to the eighth embodiment is applied will be described. Figure 41 is a fourth figure showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the learning phase.

[0245] As shown in Figure 41, the image generation system 4100 comprises a server device 110, an information processing device 120, and a server device 2540. In the image generation system 4100, the information processing device 120, the server device 110, and the server device 2540 are connected to each other via a network 150.

[0246] In the learning phase, the server device 110 includes a generating AI 111, a configuration fine-tuning unit 4111, an element fine-tuning unit 4112, and an environment fine-tuning unit 4113. When the server device 110 receives configuration learning data (an example of first learning data) from the information processing device 120 via the network 150, it uses the configuration fine-tuning unit 4111 to perform additional learning on the generating AI 111. As a result, the configuration fine-tuning unit 4111 generates a configuration-tuned model corresponding to each piece of configuration learning data.

[0247] Similarly, when the server device 110 receives element learning data (an example of second learning data) from the information processing device 120 via the network 150, it uses the element fine-tuning unit 4112 to perform additional learning on the generated AI 111. As a result, the element fine-tuning unit 4112 generates an element-tuned model corresponding to each element learning data.

[0248] Similarly, when the server device 110 receives environmental learning data (an example of third learning data) from the information processing device 120 via the network 150, it uses the environmental fine-tuning unit 4113 to perform additional learning on the generated AI 111. As a result, the environmental fine-tuning unit 4113 generates an environment-tuned model corresponding to each environmental learning data.

[0249] During the learning phase, the information processing device 120 generates training data and instructs the generating AI 111 to perform additional training. Specifically, the information processing device 120 acquires images to be included in the constructive training data from the server device 2540. The information processing device 120 generates constructive training data that includes the acquired images and captions indicating each component contained in the acquired images. The information processing device 120 transmits the generated constructive training data to the server device 110 and instructs it to perform additional training on the generating AI 111 using the generated constructive training data.

[0250] Similarly, the information processing device 120 acquires images from the server device 2540 to be included in the element learning data. The information processing device 120 generates element learning data that includes the acquired images and captions indicating each component contained in the acquired images. The information processing device 120 transmits the generated element learning data to the server device 110 and instructs it to perform additional training on the generated AI 111 using the generated element learning data.

[0251] Similarly, the information processing device 120 acquires images from the server device 2540 to be included in the environmental learning data. The information processing device 120 generates environmental learning data that includes the acquired images and captions indicating each component contained in the acquired images. The information processing device 120 transmits the generated environmental learning data to the server device 110 and instructs it to perform additional training on the generated AI 111 using the generated environmental learning data.

[0252] During the learning phase, the information processing device 120 calculates a score for each component (for each configuration learning data, each element learning data, and each environment learning data) based on a score indicating the importance of each component within the image, and stores it in association with the learning data.

[0253] The server device 2540 functions as an information provision unit 2541 and provides images to be included in the configuration learning data, element learning data, and environment learning data, respectively, to the information processing device 120 via the network 150.

[0254] <Specific examples of training data> This section describes specific examples of training data for each category (here, constructive training data, elemental training data, and environmental training data) generated by the information processing device 120 during the learning phase. Figure 42 shows specific examples of training data for each category.

[0255] As shown in Figure 42, the configured training data 4210 includes training data (for sunsets), training data (for mountains), training data (for beaches), training data (for parks), etc.

[0256] The element training data 4220 includes training data (for clouds), training data (for trees), training data (for flowers), training data (for rivers), etc.

[0257] Environmental learning data 4230 includes learning data (for temperature), learning data (for humidity), learning data (for wind speed), learning data (for rainfall), etc.

[0258] The items included in each training data set are the same regardless of the type of training data, and as explained using Figure 28, they include "image data" and "captions." In addition, each training data set is associated with a "score for each component."

[0259] <System configuration of the image generation system> The system configuration in the generation phase of an image generation system to which the information processing device according to the eighth embodiment is applied will be described. Figure 43 is the fifth figure showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the generation phase.

[0260] As shown in Figure 43, the image generation system 4300 comprises a server device 110, an information processing device 120, and a server device 140. In the image generation system 4300, the information processing device 120 and the server devices 110 and 140 are communicated with each other via a network 150.

[0261] In the generation phase, the server device 110 has a generation AI 111 and tuned models for each category (configuration-tuned model 4311, element-tuned model 4312, and environment-tuned model 4313). When the server device 110 receives a prompt from the information processing device 120 via the network 150, it operates the generation AI 111 using the configuration-tuned model 4311. As a result, the server device 110 generates a first image as an image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0262] When the server device 110 receives a prompt containing the first image from the information processing device 120 via the network 150, it operates the generation AI 111 using the element-tuned model 4312. As a result, the server device 110 generates a second image as the image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0263] When the server device 110 receives a prompt containing a second image from the information processing device 120 via the network 150, it operates the generation AI 111 using the environment-tuned model 4313. As a result, the server device 110 generates a third image as the image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0264] The information processing device 120 is the same as the information processing device 120 described with reference to Figure 31 in the fourth embodiment above, so its description is omitted here. However, in the case of the information processing device 120 according to the eighth embodiment, when sending a prompt to the server device 110, • Using the configuration-tuned model 4311, the generated AI111 is run, prompting it to generate the first image, A prompt containing a first image, which instructs the system to generate a second image by running the generation AI111 using the element-tuned model 4312, A prompt including a second image, which instructs the generation AI111 to generate a third image by running it using an environment-tuned model. Send them in order.

[0265] <Flowchart of image generation process by the image generation system> The image generation process flow in the generation phase by the image generation system 4300 will be explained. Figure 44 is the fifth flowchart showing the image generation process flow by the image generation system. Note that the difference from the first flowchart shown in Figure 17 is in steps S4401 to S4404.

[0266] In step S4401, the information processing device 120 extracts components from the text contained in the generated prompt that correspond to the component-specific scores associated with each training data. The information processing device 120 calculates the sum of the scores of each component for each training data (total score) from the scores corresponding to the extracted components. Based on the total score of each training data, the information processing device 120 then... Specify the LoRA execution for the configured and tuned model 4311. Specify the execution LoRA for the element-tuned model 4312. Specify the environment-tuned model 4313 for execution of LoRA.

[0267] In step S4402, the information processing device 120 sends a prompt to the server device 110 that includes specifying the execution LoRA of the configuration-tuned model 4311. As a result, the server device 110 operates the generation AI using the execution LoRA of the configuration-tuned model 4311 to generate the first image.

[0268] In step S4403, the information processing device 120 sends a prompt to the server device 110 that includes the generated first image and a specification of the execution LoRA of the element-tuned model 4312. As a result, the server device 110 runs the generation AI using the execution LoRA of the element-tuned model 4312 and generates a second image.

[0269] In step S4404, the information processing device 120 sends a prompt to the server device 110 that includes the generated second image and the execution LoRA of the environment-tuned model 4313. As a result, the server device 110 operates the generation AI using the execution LoRA of the environment-tuned model 4313 to generate a third image.

[0270] <Summary> As is clear from the above explanation, the information processing device 120 according to the eighth embodiment, in the learning phase, • Recognize each component contained in the collected image and calculate the confidence level of each recognized component. The collected images are classified into multiple groups according to the type of image content, and a component-specific score is calculated for each component contained in the images belonging to each group. • For each group, training data is generated in which images and captions describing each component contained in the images are associated, and component-specific scores are associated with each piece of training data. The generated training data is categorized into constructive training data, elemental training data, and environmental training data. • By using the configuration training data and the corresponding LoRA in the configuration fine-tuning unit, additional training is performed on the generative AI to generate a configuration-tuned model. • By using element training data and the corresponding LoRA in the element fine-tuning section, additional training is performed on the generative AI to generate an element-tuned model. • By using environmental training data and the corresponding LoRA in the environmental fine-tuning unit, additional training is performed on the generative AI to generate an environment-tuned model.

[0271] Furthermore, in the generation phase, the information processing apparatus 120 according to the eighth embodiment performs the following: • Environmental language is generated by acquiring environmental data of the target environment and converting it into natural language that includes sensory expressions. • Generate prompts containing text, sentences, or images that represent images co-occurring from the environment language or environment data. Based on the text elements contained in the generated prompt, specify the execution LoRA for each category. • Send a prompt including the LoRA specification for running the configuration-tuned model. This will run the generative AI using the LoRA of the configuration-tuned model to acquire the first image. • Send a prompt specifying the LoRA for running the element-tuned model and the acquired first image. This will run the generative AI using the element-tuned model's LoRA and acquire a second image. • Send a prompt specifying the LoRA location for the environment-tuned model and the acquired second image. This will run the generative AI using the environment-tuned model's LoRA and acquire a third image. The acquired third image is used to represent the air quality in the target environment and is displayed to the user.

[0272] Thus, in the information processing device 120 according to the eighth embodiment, the generating AI, which has undergone additional learning for each category (configuration, element, environment), is operated in the order of categories (configuration, element, environment). As a result, the information processing device 120 according to the eighth embodiment can generate images that intuitively represent the state of the air more appropriately.

[0273] In the above description, we listed configuration, elements, and environment as categories, but the number and types of categories are not limited to these. Also, in the above description, we assumed that one execution LoRA is specified for each category, but the number of execution LoRAs specified for each category is not limited to one, and may be multiple. For example, as described in the sixth embodiment, if it is determined that the text included in the prompt does not satisfy the environmental conditions attached to the training data, the execution LoRA may be respecified, and as a result, multiple execution LoRAs may be specified for a particular category.

[0274] [Ninth Embodiment] Specific examples of images that intuitively represent the state of air, generated by an image generation system including the information processing device 120 according to the sixth and eighth embodiments described above, will be explained. Figure 45 is a diagram showing an example of image generation processing by the image generation system.

[0275] In the example shown in Figure 45, the information processing device 120 uses environmental data, • Date and time: July 25th, 4 PM • Location: Karuizawa ·Temperature: 25℃ ·Humidity: 60% ·Air volume: 2m / s This shows how the data was obtained.

[0276] The example in Figure 45 shows how the information processing device 120 generates environmental language such as "hot," "relatively clear visibility," "moderately strong wind," "sunset," and "summer resort" based on environmental data.

[0277] The example in Figure 45 shows how the information processing device 120 generates the following sentence to be included in the prompt, based on the environmental language and environmental data: "A summer evening in Karuizawa. The sound of cicadas can be heard under the intense sunlight. Stepping into the shade of a tree, a refreshing breeze blows through, slightly alleviating the heat."

[0278] The example in Figure 45 shows that the information processing device 120, based on environmental data, includes the following image editing parameters in the prompt: Brightness: +15 Contrast: +10 Saturation: +15 Sharpness: +10 • Blur: 2-3 This shows how they identified it.

[0279] The example in Figure 45 shows that the information processing device 120 is • Specify the sunset LoRA as the LoRA to run the configured and tuned model. Instead of running the environment-tuned model's LoRA, specify the element-tuned model's LoRA for wind speed. • Run the element-tuned model, specifying the temperature LoRA as the LoRA. It shows the situation.

[0280] In the example shown in Figure 45, reference numerals 4501 to 4503 indicate images obtained when the information processing device 120 operates the generating AI by separately including the execution LoRA for each specified category in the prompt, for reference.

[0281] On the other hand, in the example in Figure 45, reference numeral 4510 indicates an image obtained when the information processing device 120 operated the generating AI by sequentially including the specification of execution LoRA for each category in the prompt. As is clear from the comparison between the images shown by reference numerals 4501 to 4503 and the image shown by reference numeral 4510, • When each category's LoRA is run individually to generate images, • When images are generated by sequentially running each category's LoRA execution, Therefore, the resulting images will be significantly different.

[0282] In this way, by classifying multiple tuned models into categories and operating them in the appropriate order, it is possible to generate images that intuitively represent the state of the air with high accuracy.

[0283] [Other embodiments] In each of the above embodiments, it has been described that the information processing program is executed by a single information processing device 120, but the information processing program may be executed by multiple information processing devices working together.

[0284] The functions realized by the information processing device 120 by executing the information processing program are not limited to the functions described in each of the above embodiments. For example, some or all of the functions of the server device 110, or some or all of the functions of the server device 140, may be realized in the information processing device 120.

[0285] Although embodiments have been described above, it should be understood that various modifications to the form and details are possible without departing from the spirit and scope of the claims. [Explanation of Symbols]

[0286] 100: Image generation system 110: Server device 111: Generation AI 120: Information Processing Device 130_1~130_n: Sensor 1~Sensor n 200: Control Unit 310: Communication Control Unit 320: Environmental data acquisition unit 330:Environmental language generation section 340: Prompt generation unit 350: Output section 2110: Operation control language generation unit 2500: Image generation system 2511: Fine Tuning Department 2540: Server device 2610: Image Data Collection Department 2620: Caption generation unit 2630: Training data generation unit 2640: Additional Learning Section 3100: Image generation system 3110: Tuned Model 3210: Weight calculation unit 3610: Selection section 3810: Weight calculation unit 4100: Image generation system 4111: Configuration Fine Tuning Section 4112: Element Fine Tuning Section 4113: Environmental Fine Tuning Department 4210: Data for constructing and learning 4220: Data for element training 4230: Environmental learning data 4300: Image generation system 4311: Configuration-tuned model 4312: Element-tuned model 4313: Environment-tuned model

Claims

1. An information processing device having a control unit, The control unit, By acquiring environmental data from the target environment and converting it into natural language that includes sensory expressions, an environmental language is generated. A prompt is generated that includes words or sentences representing images that co-occur from the aforementioned environmental language. Based on the generated prompt, select a generative AI that has been pre-trained using multiple training datasets, and operate it based on the assigned weights. The system displays an image showing the air quality of the target environment, generated by the aforementioned AI. Information processing device.

2. The aforementioned generating AI is further trained using a model for generating specific images and corresponding training data. The corresponding training data includes a specific set of images and captions indicating each component contained in each specific image. The information processing apparatus according to claim 1.

3. The training data corresponding to the model for generating the aforementioned specific images is associated with component-specific scores calculated when recognizing each component contained in each of the aforementioned specific images, based on the confidence level of each component. The information processing apparatus according to claim 2.

4. The control unit, The system extracts each component from the generated prompt, and then calculates the total score for each training dataset by summing the scores of each extracted component based on the component-specific scores associated with the training data. The information processing apparatus according to claim 1.

5. The control unit, Based on the aforementioned total score, select the training data. When running the generative AI that has been further trained using the model corresponding to the selected training data, the weights of the corresponding model are calculated based on the total score. The information processing apparatus according to claim 4.

6. The control unit, Based on the aforementioned total score, select the training data. Using a large-scale language model, the importance of the model's prompts corresponding to the selected training data is determined. Based on the determined importance, the weights of the model are calculated when the generative AI, which has been further trained using the model corresponding to the selected training data, is operated. The information processing apparatus according to claim 4.

7. The control unit, Based on the aforementioned total score, select the training data. The environmental conditions attached to the selected training data are determined. The information processing apparatus according to claim 4.

8. The control unit, If environmental conditions are attached to the selected training data, and the text included in the prompt satisfies the environmental conditions, the generative AI, which has been further trained using the model corresponding to the selected training data, is operated based on predetermined weights. If environmental conditions are attached to the selected training data, and the text included in the prompt does not satisfy the environmental conditions, then a generative AI that has been further trained using a model other than the model corresponding to the selected training data is operated based on predetermined weights. If no environmental conditions are attached to the selected training data, the generative AI, which has been further trained using a model corresponding to arbitrary training data, is operated based on predetermined weights. The information processing apparatus according to claim 7.

9. The control unit, Based on the aforementioned total score, select the training data. When operating the generative AI that has been further trained using the model corresponding to the selected training data, multiple combinations of the model's weights are obtained. The generating AI, which has been further trained using the model corresponding to the selected training data, is operated based on each combination of weights to identify the combination of weights corresponding to the image evaluated by the user from among the images generated by the generating AI. The information processing apparatus according to claim 4.

10. The aforementioned training data is The first training data for further learning the image composition, Second training data for further learning of specific elements contained in the image, A third training dataset for further learning about the environment represented by the image, Includes, The control unit, By operating the generative AI that has been further trained using the first training data, the first image generated by the generative AI is acquired. By running the generative AI, which has been further trained using the second training data, on the first image, the second image generated by the generative AI is obtained. By running the generative AI, which has been further trained using the third training data, with the second image, a third image generated by the generative AI is obtained. The acquired third image is displayed as an image showing the air quality of the target environment. The information processing apparatus according to claim 1.

11. An information processing device having a control unit, The control unit, Recognize each component contained in each specific image, Using training data that includes a specific set of images and captions indicating each component recognized in each specific image, a generative AI is further trained using a model for generating specific images. The score for each component, calculated when recognizing each component contained in each of the aforementioned specific images, according to the confidence level of each component, is stored in association with the training data. Information processing device.

12. The aforementioned training data is The first training data for further learning the image composition, Second training data for further learning of specific elements contained in the image, A third training dataset for further learning about the environment represented by the image, including, The information processing apparatus according to claim 11.

13. The control unit of the information processing device, By acquiring environmental data from the target environment and converting it into natural language that includes sensory expressions, an environmental language is generated. A prompt is generated that includes words or sentences representing images that co-occur from the aforementioned environmental language. Based on the generated prompt, select a generative AI that has been pre-trained using multiple training datasets, and operate it based on the assigned weights. The system displays an image showing the air quality of the target environment, generated by the aforementioned AI. An information processing method that performs a process.

14. The control unit of the information processing device, Recognize each component contained in each specific image, Using training data that includes a specific set of images and captions indicating each component contained in each specific image, a generative AI is further trained using a model for generating specific images. The score for each component, calculated when recognizing each component contained in each of the aforementioned specific images, according to the confidence level of each component, is stored in association with the training data. An information processing method that performs a process.

15. The control unit of the information processing device has By acquiring environmental data from the target environment and converting it into natural language that includes sensory expressions, an environmental language is generated. A prompt is generated that includes words or sentences representing images that co-occur from the aforementioned environmental language. Based on the generated prompt, select a generative AI that has been pre-trained using multiple training datasets, and operate it based on the assigned weights. The system displays an image showing the air quality of the target environment, generated by the aforementioned AI. An information processing program used to execute a process.

16. The control unit of the information processing device has Recognize each component contained in each specific image, Using training data that includes a specific set of images and captions indicating each component contained in each specific image, a generative AI is further trained using a model for generating specific images. The score for each component, calculated when recognizing each component contained in each of the aforementioned specific images, according to the confidence level of each component, is stored in association with the training data. An information processing program used to execute a process.

Images