Information processing device, information processing method, and information processing program

Abstract

The present invention generates an image in which a state of the air is sensorily expressed. This information processing device includes a control unit. The control unit acquires environment data including an item value of an item indicating a state of the air in an environment to be processed or an item value of an item affecting the state of the air in the environment to be processed, generates an environment language by converting the environment data into a natural language including a sensory expression, generates a prompt including a word, a sentence, or an image representing an image co-occurring from the environment language, operates a generation AI by using the prompt, and displays an image generated by the generation AI and indicating the state of the air in the environment to be processed.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] There is known a technique for visualizing environmental data such as temperature, humidity, wind speed, and carbon dioxide concentration. This technique makes it possible to quantitatively grasp, for example, the state of the air, such as temperature, humidity, wind speed, and carbon dioxide concentration.

[0003] JP 2024-60907 A

[0004] On the other hand, until now, there has been no means to intuitively express the state of air (for example, a means to visualize human sensations). For example, even using generative AI, it has been difficult to directly generate an image that intuitively expresses the state of air from the quantitative data described above.

[0005] The present disclosure aims to generate an image that intuitively expresses the state of air.

[0006] A first aspect of the present disclosure is an information processing device having a control unit, wherein the control unit acquires environmental data including item values ​​of items indicating the air condition of a target environment or item values ​​of items that affect the air condition of the target environment, generates environmental language by converting the environmental data into natural language including sensory expressions, generates prompts including words, sentences, or images that express images that co-occur from the environmental language, operates a generation AI using the prompt, and displays an image generated by the generation AI that indicates the air condition of the target environment.

[0007] A second aspect of the present disclosure is the information processing device according to the first aspect, wherein the control unit generates the prompt including a phrase, a sentence, or an image that expresses an image that co-occurs from the environmental data.

[0008] A third aspect of the present disclosure is an information processing device according to the first or second aspect, wherein the control unit acquires image editing parameters corresponding to the environmental data or image editing parameters corresponding to a sentence included in the prompt, and generates the prompt including the image editing parameters.

[0009] A fourth aspect of the present disclosure is an information processing device according to any one of the first to third aspects, wherein the control unit acquires image editing parameters according to the identified style, and generates the prompt including the image editing parameters.

[0010] A fifth aspect of the present disclosure is an information processing device described in any of the first to third aspects, wherein the control unit, when displaying an image showing the air condition of the target environment, displays an image showing the air condition of a comparison environment different from the target environment in comparison therewith.

[0011] A sixth aspect of the present disclosure is an information processing device described in any of the first to fifth aspects, wherein the control unit generates a motion control language representing the movement of components based on the environmental data, generates a prompt including the motion control language, operates a generation AI using the prompt, and displays a moving image generated by the generation AI showing the air state of the target environment, in which the components move according to the motion control language.

[0012] A seventh aspect of the present disclosure is an information processing method, in which a control unit of an information processing device performs the following processing: acquires environmental data including item values ​​of items indicating the air condition of a target environment or item values ​​of items that affect the air condition of the target environment; generates environmental language by converting the environmental data into natural language including sensory expressions; generates prompts including words, sentences, or images that express images that co-occur from the environmental language; operates a generation AI using the prompt; and displays an image generated by the generation AI that indicates the air condition of the target environment.

[0013] An eighth aspect of the present disclosure is an information processing program that causes a control unit of an information processing device to perform the following process: acquire environmental data including item values ​​of items indicating the air condition of a target environment or item values ​​of items that affect the air condition of the target environment; generate environmental language by converting the environmental data into natural language including sensory expressions; generate prompts including words, sentences, or images that express images that co-occur from the environmental language; operate a generation AI using the prompts; and display an image generated by the generation AI that indicates the air condition of the target environment.

[0014] FIG. 1 is a first diagram showing an example of a system configuration of an image generation system. FIG. 2 is a diagram showing an example of a hardware configuration of an information processing device. FIG. 3 is a first diagram showing an example of a functional configuration of an information processing device. FIG. 4 is a diagram showing a specific example of processing by an environmental data acquisition unit. FIG. 5 is a first diagram showing a specific example of processing by an environmental language generation unit. FIG. 6 is a second diagram showing a specific example of processing by an environmental language generation unit. FIG. 7 is a third diagram showing a specific example of processing by an environmental language generation unit. FIG. 8 is a first diagram showing a specific example of processing by a prompt generation unit. FIG. 9 is a second diagram showing a specific example of processing by a prompt generation unit. FIG. 10 is a third diagram showing a specific example of processing by a prompt generation unit. FIG. 11 is a fourth diagram showing a specific example of processing by a prompt generation unit. FIG. 12 is a fifth diagram showing a specific example of processing by a prompt generation unit. FIG. 13 is a sixth diagram showing a specific example of processing by a prompt generation unit. FIG. 14 is a seventh diagram showing a specific example of processing by a prompt generation unit. FIG. 15 is an eighth diagram showing a specific example of processing by the prompt generation unit. FIG. 16 is a ninth diagram showing a specific example of processing by the prompt generation unit. FIG. 17 is a first flowchart showing the flow of image generation processing by the image generation system. FIG. 18 is a first diagram showing an example of a display screen. FIG. 19 is a second diagram showing an example of the functional configuration of an information processing device. FIG. 20 is a second diagram showing an example of a display screen. FIG. 21 is a third diagram showing an example of the functional configuration of an information processing device. FIG. 22 is a diagram showing a specific example of processing by the action control language generation unit. FIG. 23 is a diagram showing an example of a prompt (sentence) and an example of a moving image. FIG. 24 is a second flowchart showing the flow of image generation processing by the image generation system. FIG. 25 is a second diagram showing an example of the system configuration of an image generation system. FIG. 26 is a fourth diagram showing an example of the functional configuration of an information processing device. FIG. 27 is a diagram showing a specific example of processing by the caption generation unit. FIG. 28 is a diagram showing a specific example of processing by the learning data generation unit. Fig. 29 is a diagram showing details of the generation AI and fine tuning unit. Fig. 30 is a flowchart showing the flow of additional learning processing by the image generation system. Fig. 31 is a third diagram showing an example of the system configuration of the image generation system.FIG. 32 is a fifth diagram showing an example of the functional configuration of an information processing device. FIG. 33 is a first diagram showing a specific example of processing by the weight calculation unit. FIG. 34 is a third flowchart showing the flow of image generation processing by the image generation system. FIG. 35 is a second diagram showing a specific example of processing by the weight calculation unit. FIG. 36 is a sixth diagram showing an example of the functional configuration of an information processing device. FIG. 37 is a diagram showing a specific example of processing by the selection unit. FIG. 38 is a seventh diagram showing an example of the functional configuration of an information processing device. FIG. 39 is a third diagram showing a specific example of processing by the weight calculation unit. FIG. 40 is a fourth flowchart showing the flow of image generation processing by the image generation system. FIG. 41 is a fourth diagram showing an example of the system configuration of an image generation system. FIG. 42 is a diagram showing specific examples of learning data for each category. FIG. 43 is a fifth diagram showing an example of the system configuration of an image generation system. FIG. 44 is a fifth flowchart showing the flow of image generation processing by the image generation system. FIG. 45 is a diagram showing an example of image generation processing by the image generation system.

[0015] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0016] [First embodiment] <System configuration of image generation system> A system configuration of an image generation system to which an information processing apparatus according to a first embodiment is applied will be described. Fig. 1 is a first diagram showing an example of the system configuration of an image generation system.

[0017] 1 , the image generation system 100 includes a server device 110, an information processing device 120, and a server device 140. In the image generation system 100, the information processing device 120, the server device 110, and the server device 140 are communicatively connected via a network 150.

[0018] The server device 110 has a generation AI 111, and when it receives a prompt from the information processing device 120 via the network 150, it operates the generation AI 111 to generate an image (a still image or a moving image) according to the prompt. The server device 110 transmits the generated image to the information processing device 120 via the network 150.

[0019] The information processing device 120 is a device that displays an image that sensorily expresses the air condition in a target environment. In the first embodiment, the information processing device 120 causes the generation AI 111 to generate an image (still image) that sensorily expresses the air condition in the target environment, and acquires and displays the image from the generation AI 111. Note that the target environment refers to the space that is the target when expressing the air condition. The information processing device 120 executes various processes required for the generation AI 111 to generate an image.

[0020] Specifically, the information processing device 120 acquires: Environmental data of the target environment (e.g., data on temperature, humidity, etc.) measured by sensors 1 to n (reference numerals 130_1 to 130_n) installed in the target environment, or Environmental data of the target environment (e.g., data on season, location, etc.) input by the user. The information processing device 120 acquires the environmental data of the target environment (e.g., data on rainfall, wind speed, etc.) from the server device 140 via the network 150.

[0021] The information processing device 120 generates environmental language by converting the acquired environmental data of the target environment into natural language including sensory expressions. The information processing device 120 generates a prompt including a phrase, sentence, or image expressing an image co-occurring from either or both of the environmental data and the environmental language. The information processing device 120 operates a generation AI using the generated prompt and displays an image indicating the air condition of the target environment, generated by the generation AI.

[0022] The server device 140 functions as an information provider 141 and provides environmental data of the target environment (e.g., data on rainfall, wind speed, etc.) to the information processing device 120 via the network 150. For example, the server device 140 acquires location information (latitude, longitude, altitude) of the target environment from the information processing device 120, and collects environmental data corresponding to the location information at predetermined time intervals. The server device 140 provides the collected environmental data to the information processing device 120 at predetermined time intervals.

[0023] 1 illustrates a case where the information processing device 120 is installed outside the target environment, but the information processing device 120 may be installed within the target environment. Also, the example of Fig. 1 does not mention a user viewing an image showing the air condition of the target environment displayed by the information processing device 120, but the user may view the image either within the target environment or outside the target environment.

[0024] <Hardware configuration of information processing device> The hardware configuration of the information processing device 120 will be described. Fig. 2 is a diagram showing an example of the hardware configuration of the information processing device. As shown in Fig. 2, the information processing device 120 has a processor 201, a memory 202, an auxiliary storage device 203, a user interface device 204, a communication device 205, a connection device 206, and a drive device 207. Note that the respective hardware components of the information processing device 120 are connected to each other via a bus 208.

[0025] The processor 201 has various computing devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 201 reads various programs (for example, information processing programs, etc.) into the memory 202 and executes them.

[0026] The memory 202 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 201 and the memory 202 form a so-called computer (also referred to as a "control unit 200"), and the processor 201 executes various programs read onto the memory 202, causing the computer to realize various functions.

[0027] The auxiliary storage device 203 stores various programs and various information used when the processor 201 executes the various programs.

[0028] The user interface device 204 includes an operation device for inputting user instructions and a display device for displaying a display screen including an image.

[0029] The communication device 205 is a device that is connected to the network 150 and performs communication processing with the server devices 110, 140, and the like.

[0030] The connection device 206 is a device that connects the sensors 1 to n (reference numerals 130_1 to 130_n) and the information processing device 120.

[0031] The drive device 207 is a device for loading a recording medium 210. The recording medium 210 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The recording medium 210 may also include semiconductor memory that records information electrically, such as a ROM, a flash memory, etc.

[0032] The various programs to be installed in the auxiliary storage device 203 are installed, for example, by setting the distributed recording medium 210 in the drive device 207 and reading the various programs recorded on the recording medium 210 by the drive device 207. Alternatively, the various programs to be installed in the auxiliary storage device 203 may be installed by being downloaded from the network 150 via the communication device 205.

[0033] <Functional Configuration of Information Processing Device> The functional configuration of the information processing device 120 will be described. Fig. 3 is a first diagram showing an example of the functional configuration of the information processing device. As described above, an information processing program is installed in the information processing device 120, and by executing the information processing program, the information processing device 120 functions as: a communication control unit 310; an environmental data acquisition unit 320; an environmental language generation unit 330; a prompt generation unit 340; and an output unit 350.

[0034] The communication control unit 310 acquires environmental data from the server device 140 via the network 150 and notifies the environmental data acquisition unit 320. The communication control unit 310 acquires the prompt generated by the prompt generation unit 340 and transmits it to the server device 110 via the network 150, and also receives an image from the server device 110 and notifies the output unit 350.

[0035] The environmental data acquisition unit 320 acquires the following: environmental data notified from the communication control unit 310; environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n); and environmental data input by the user, and notifies the environmental language generation unit 330 and the prompt generation unit 340 of the same.

[0036] The environmental language generation unit 330 generates environmental language based on the environmental data notified by the environmental data acquisition unit 320 by referencing the conversion table stored in the conversion table storage unit 360. Environmental language is natural language that intuitively expresses the state of the air in the target environment. "Sensuous expression" refers to putting into words the sensations that a person actually in the target environment would feel. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0037] The prompt generation unit 340 generates a prompt including: a phrase, a sentence, or an image that expresses an image that co-occurs from either or both of the environmental data notified by the environmental data acquisition unit 320 and the environmental language notified by the environmental language generation unit 330; and image editing parameters according to the environmental data notified by the environmental data acquisition unit 320. The prompt generation unit 340 notifies the communication control unit 310 of the generated prompt, and also notifies the output unit 350. Note that the image editing parameters refer to quantitative indicators that affect image quality, and specific examples will be described later.

[0038] The output unit 350 generates a display screen based on the image notified by the communication control unit 310 and the wording or sentence, image editing parameters, etc. contained in the prompt notified by the prompt generation unit 340, and displays it to the user.

[0039] <Specific Examples of Processing by Each Unit of Information Processing Apparatus> A specific example of processing by each unit of the information processing apparatus 120 (here, the environment data acquisition unit 320, the environment language generation unit 330, and the prompt generation unit 340) will be described.

[0040] (1) Specific Example of Processing by the Environmental Data Acquisition Unit Fig. 4 is a diagram showing a specific example of processing by the environmental data acquisition unit. In Fig. 4, reference numeral 401 denotes an example of environmental data provided by the server device 140 and acquired by the environmental data acquisition unit 320 from the communication control unit 310. As indicated by reference numeral 401 in Fig. 4, the environmental data provided by the server device 140 includes rainfall, wind direction, wind speed, weather, a discomfort index, etc.

[0041] 4, reference numeral 402 denotes an example of environmental data acquired by the environmental data acquisition unit 320 through measurements made by sensors 1 to n (reference numerals 130_1 to 130_n) or input by a user. As indicated by reference numeral 402 in Fig. 4, the environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n) or input by a user includes temperature, humidity, date, time, carbon dioxide concentration, season, location, etc.

[0042] The environmental data acquisition unit 320 notifies the acquired environmental data to the environmental language generation unit 330. When notifying the environmental data, the environmental data acquisition unit 320 may separate each item included in the environmental data into an item indicating the air condition of the target environment and an item that affects the air condition of the target environment.

[0043] The example of Fig. 4 shows how temperature, humidity, discomfort index, rainfall, wind direction, wind speed, carbon dioxide concentration, etc. are notified to the environmental language generation unit 330 as items indicating the air condition of the target environment. The example of Fig. 4 also shows how date, time, season, weather, location, etc. are notified to the environmental language generation unit 330 as items that affect the air condition of the target environment. As shown in Fig. 4, the item values ​​of each item indicating the air condition of the target environment and the item values ​​of each item that affects the air condition of the target environment may be notified to the environmental language generation unit 330 at predetermined intervals, for example.

[0044] (2) Specific Example of Processing by the Environmental Language Generation Unit Fig. 5 is a first diagram showing a specific example of processing by the environmental language generation unit. As shown in Fig. 5, the conversion table storage unit 360 stores a conversion table in which item values ​​of multiple items indicating the air condition of the target environment among the environmental data are associated with environmental language, which is natural language including sensory expressions.

[0045] Of these, conversion table 510 is a table in which each item value of the item = "temperature" is associated with an environmental language. Conversion table 520 is a table in which each item value of the item = "humidity" is associated with an environmental language. Conversion table 530 is a table in which each item value of the item = "discomfort index" is associated with an environmental language. Conversion table 540 is a table in which each item value of the item = "rainfall" is associated with an environmental language. Conversion table 550 is a table in which each item value of the item = "wind speed" is associated with an environmental language. Note that when each item value indicating the air condition of the target environment is included, as in conversion tables 510 to 550, a conversion table that takes into account compatibility with bodily sensations may be created by: - ​​logarithmically converting each item value, or - approximating each item value with a sigmoid function.

[0046] When the environmental language generation unit 330 is notified of the item values ​​of items indicating the air condition of the target environment, such as temperature, humidity, discomfort index, rainfall, and wind speed, as environmental data from the environmental data acquisition unit 320, the environmental language generation unit 330 refers to the conversion tables 510 to 550. As a result, the environmental language generation unit 330 converts the item values ​​of items indicating the air condition of the target environment, such as temperature, humidity, discomfort index, rainfall, and wind speed, into natural language including sensory expressions, and generates environmental language. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0047] 6 is a second diagram showing a specific example of processing by the environmental language generation unit. As shown in Fig. 6, the conversion table storage unit 360 stores a conversion table in which item values ​​of multiple items that affect the air condition of the target environment are associated with environmental language, which is natural language including sensory expressions.

[0048] Of these, the conversion table 610 is a table in which each item value of the item="date" is associated with the environment language.

[0049] When the environmental language generation unit 330 receives environmental data from the environmental data acquisition unit 320, such as the date, which is an item value that affects the air quality of the target environment, the environmental language generation unit 330 refers to the conversion table 610. The environmental language generation unit 330 converts the date, which is an item value that affects the air quality of the target environment, into natural language that includes sensory expressions, and generates environmental language. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0050] 7 is a third diagram showing a specific example of processing by the environmental language generation unit. As shown in Fig. 7, the conversion table storage unit 360 stores a conversion table in which combinations of item values ​​of multiple items that affect the air condition of the target environment are associated with environmental languages, which are natural languages ​​including sensory expressions.

[0051] Of these, conversion table 710 is a table in which combinations of item values ​​for the items "season", "date", and "location" are associated with environment languages. Similarly, conversion table 720 is a table in which combinations of item values ​​for the items "season", "date", and "location" are associated with environment languages. Conversion table 710 and conversion table 720 have the same items but different item values. Therefore, the associated environment languages ​​are different.

[0052] When the environmental language generation unit 330 receives environmental data from the environmental data acquisition unit 320, such as a combination of item values ​​of items that affect the air quality of the target environment, such as season, date, and location, the environmental language generation unit 330 refers to the conversion tables 710 to 720. The environmental language generation unit 330 converts the combination of item values ​​of items that affect the air quality of the target environment, such as season, date, and location, into natural language that includes sensory expressions, thereby generating an environmental language. The environmental language generation unit 330 notifies the prompt generation unit 340 of the generated environmental language.

[0053] (3) Specific Example of Processing by the Prompt Generator Figure 8 is the first diagram showing a specific example of processing by the prompt generator. As described above, the prompt generator 340 generates a prompt including a sentence expressing an image that co-occurs from the environmental data (here, the item values ​​of the items that affect the air quality of the target environment) notified from the environmental data acquisition unit 320, and notifies the communication control unit 310 of the prompt.

[0054] 8 shows how the prompt generation unit 340 generates sentences expressing images that co-occur with the respective item values ​​when the communication control unit 310 notifies the communication control unit 310 of the environmental data, such as the item values ​​that affect the air quality of the target environment. The prompt generation unit 340 then notifies the communication control unit 310 of a prompt that includes the generated sentences.

[0055] 9 is a second diagram showing a specific example of processing by the prompt generation unit 340. As described above, the prompt generation unit 340 generates a prompt including a sentence expressing an image that co-occurs from: environmental data notified from the environmental data acquisition unit 320 (here, item values ​​of items that affect the air quality of the target environment); and environmental language notified from the environmental language generation unit 330, and notifies the communication control unit 310 of the generated prompt.

[0056] 9 shows the state in which the prompt generation unit 340 generates a sentence expressing an image that co-occurs from the environmental data and the environmental language in the following cases: The environmental data acquisition unit 320 notifies the user of item values ​​for items that affect the air quality of the target environment, such as "location," "season," and "time," as environmental data, and the environmental language generation unit 330 notifies the user of "hot" as environmental language. The prompt generation unit 340 then notifies the communication control unit 310 of a prompt that includes the generated sentence.

[0057] 10 is a third diagram illustrating a specific example of processing by the prompt generation unit 340. As described above, the prompt generation unit 340 generates a prompt including a sentence expressing an image that co-occurs from: environmental data notified from the environmental data acquisition unit 320 (here, item values ​​of items that affect the air quality of the target environment); and environmental language notified from the environmental language generation unit 330, and notifies the communication control unit 310 of the generated prompt.

[0058] 10 shows how the prompt generation unit 340 generates a sentence expressing an image that co-occurs from the environmental data and the environmental language in the following cases: The environmental data acquisition unit 320 notifies the user of item values ​​for items that affect the air quality of the target environment, such as "location" and "time," as environmental data, and the environmental language generation unit 330 notifies the user of "very hot" as environmental language. The prompt generation unit 340 then notifies the communication control unit 310 of a prompt that includes the generated sentence.

[0059] 11 is a fourth diagram illustrating a specific example of processing by the prompt generation unit 340. As described above, the prompt generation unit 340 generates a prompt including an image representing an image that co-occurs from: environmental data notified from the environmental data acquisition unit 320 (here, item values ​​of items that affect the air quality of the target environment); and environmental language notified from the environmental language generation unit 330, and notifies the communication control unit 310 of the generated prompt.

[0060] 11 shows the example in which the prompt generation unit 340 generates an image that represents an image that co-occurs from the environmental data and the environmental language in the following cases: the environmental data acquisition unit 320 notifies the user of item values ​​for items that affect the air quality of the target environment, such as "location" and "time," as environmental data, and the environmental language generation unit 330 notifies the user of "very hot" as environmental language. The prompt generation unit 340 then notifies the communication control unit 310 of a prompt that includes the generated image.

[0061] 12 is a fifth diagram illustrating a specific example of processing by the prompt generation unit 340. As described above, the prompt generation unit 340 adjusts image editing parameters based on the environmental data (here, the item values ​​of items that affect the air quality of the target environment) notified from the environmental data acquisition unit 320, includes the adjusted image editing parameters in a prompt, and notifies the communication control unit 310 of the adjusted image editing parameters.

[0062] The example of Figure 12 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the user of environmental data such as "season" or "time of day" that affects the air quality of the target environment. As shown in the example of Figure 12, when the environmental data acquisition unit 320 notifies the user of environmental data such as "season" or "time of day" that affects the air quality of the target environment, the prompt generation unit 340 adjusts the image editing parameters such as "brightness," "contrast," "saturation," and "hue." The prompt generation unit 340 also adjusts the image editing parameters such as "filter," "blur," "sharpness," "shadow," and "noise reduction."

[0063] 13 is a sixth diagram illustrating a specific example of processing by the prompt generation unit 340. As described above, the prompt generation unit 340 adjusts image editing parameters based on the environmental data (here, the item values ​​of the items indicating the air quality of the target environment) notified from the environmental data acquisition unit 320, includes the adjusted image editing parameters in a prompt, and notifies the communication control unit 310 of the adjusted image editing parameters.

[0064] 13 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the user of the item values ​​of items indicating the air condition of the target environment, such as "temperature" or "humidity." As shown in the example of FIG. 13, when the environmental data acquisition unit 320 notifies the user of the item values ​​of items indicating the air condition of the target environment, such as "temperature" or "humidity," the prompt generation unit 340 adjusts the image editing parameters "brightness," "contrast," and "saturation." The prompt generation unit 340 also adjusts the image editing parameters "hue," "filter," "blur," "sharpness," "shadow" or "texture," "noise removal," and the like.

[0065] 14 is a seventh diagram illustrating a specific example of processing by the prompt generation unit 340. As described above, the prompt generation unit 340 adjusts the image editing parameters based on the environmental data (here, the item values ​​of the items indicating the air quality of the target environment) notified from the environmental data acquisition unit 320, includes the adjusted image editing parameters in a prompt, and notifies the communication control unit 310 of the adjusted image editing parameters.

[0066] 14 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the user of environmental data such as "wind speed" or "rainfall" that indicates the air condition of the target environment. As shown in the example of FIG. 14, when the environmental data acquisition unit 320 notifies the user of environmental data such as "wind speed" or "rainfall" that indicates the air condition of the target environment, the prompt generation unit 340 adjusts the image editing parameters "brightness," "contrast," and "saturation." The prompt generation unit 340 also adjusts the image editing parameters "hue," "filter," "blur," "sharpness," "effect" or "texture," "noise reduction," and the like.

[0067] 15 is an eighth diagram illustrating a specific example of processing by the prompt generation unit 340. As described above, the prompt generation unit 340 adjusts image editing parameters based on the environmental data (here, the item values ​​of items that affect the air quality of the target environment) notified from the environmental data acquisition unit 320, includes the adjusted image editing parameters in a prompt, and notifies the communication control unit 310 of the adjusted image editing parameters.

[0068] The example of Figure 15 shows how the prompt generation unit 340 adjusts image editing parameters based on environmental data when the environmental data acquisition unit 320 notifies the prompt generation unit 340 of the item values ​​of items that affect the air quality of the target environment, such as "location," as environmental data. As shown in the example of Figure 15, when the prompt generation unit 340 notifies the prompt generation unit 340 of the item values ​​of items that affect the air quality of the target environment, such as "location," as environmental data, the prompt generation unit 340 adjusts the image editing parameters "hue," "filter," and "texture." The prompt generation unit 340 also adjusts the image editing parameters "blur," "shadows and highlights," "noise," "brightness and contrast (or shadows)," and the like.

[0069] 16 is a ninth diagram illustrating a specific example of processing by the prompt generation unit. For example, when the prompt generation unit 340 generates a sentence to be included in a prompt, the prompt generation unit 340 further identifies an image style based on the generated sentence. Then, the prompt generation unit 340 adjusts image editing parameters based on the identified "image style."

[0070] The example in Figure 16 shows how the prompt generation unit 340 has identified the painting style as "watercolor style" and adjusted the image editing parameters such as "blur," "saturation," "add texture," "brush stroke," "hue," and "transparency adjustment."

[0071] The example in Figure 16 shows how the prompt generation unit 340 has identified the painting style as "oil painting" and has adjusted the image editing parameters such as "sharpness," "filter," "brush stroke," "color adjustment," "texture addition," and "shadows and highlights."

[0072] The example in Figure 16 shows how the prompt generation unit 340 has identified the painting style as "abstract painting" and adjusted the image editing parameters such as "saturation," "filter," "hue," "add noise," "blur and sharpness," and "layer blend mode."

[0073] Although the description here has been given of the case where the prompt generation unit 340 identifies the "art style" based on the sentence included in the prompt, the prompt generation unit 340 may also identify the "art style" based on the user's specification.

[0074] <Flow of Image Generation Processing by Image Generation System> A description will be given of the flow of image generation processing by the image generation system 100. Fig. 17 is a first flowchart showing the flow of image generation processing by the image generation system.

[0075] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0076] In step S1702, the information processing apparatus 120 generates an environmental language by converting the acquired environmental data into a natural language including sensory expressions.

[0077] In step S1703, the information processing device 120 generates a prompt based on either or both of the generated environmental language and the acquired environmental data. The prompt generated by the information processing device 120 includes: a phrase or sentence expressing an image co-occurring with either or both of the environmental data and the environmental language; an image expressing an image co-occurring with either or both of the environmental data and the environmental language; and image editing parameters according to the environmental data.

[0078] In step S1704, the information processing device 120 transmits the generated prompt to the server device 110. As a result, the generation AI 111 of the server device 110 generates an image according to the prompt.

[0079] In step S1705, the information processing device 120 acquires the image generated by the generation AI 111 of the server device 110 and displays it to the user.

[0080] In step S1706, the information processing apparatus 120 determines whether or not to end the image generation process. If it is determined in step S1706 that the image generation process should be continued (NO in step S1706), the process returns to step S1701.

[0081] On the other hand, if it is determined in step S1706 that the image generation process is to be ended (YES in step S1706), the image generation process is ended.

[0082] <Display Example> A description will be given of a display example in which the information processing device 120 displays an image generated by the server device 110 based on a prompt generated by the information processing device 120. Fig. 18 is a first diagram showing an example of a display screen.

[0083] 18 , display screen 1800 has area 1810 for displaying an image, area 1820 for displaying environmental data, area 1830 for displaying image editing parameters, and area 1840 for displaying text included in the prompt used to generate the image. Note that display screen 1800 may have an area for displaying environmental language instead of or in addition to area 1820 for displaying environmental data.

[0084] <Summary> As is clear from the above explanation, the information processing device 120 according to the first embodiment: - Acquires environmental data including item values ​​of items indicating the air condition of the target environment or item values ​​of items that affect the air condition of the target environment. - Generates environmental language by converting the environmental data into natural language including sensory expressions. - Generates prompts including words, sentences, or images that express images that co-occur from the environmental language or environmental data. - Operates a generation AI using the prompt, and displays an image indicating the air condition of the target environment, generated by the generation AI.

[0085] As a result, the information processing device 120 according to the first embodiment can generate an image that intuitively expresses the state of the air in the target environment.

[0086] Second Embodiment In the first embodiment, an image intuitively representing the air condition of a target environment is displayed on the display screen 1800. However, the image displayed on the display screen is not limited to an image intuitively representing the air condition of the target environment. For example, an image intuitively representing the opposite air condition (air condition of a comparison environment) to the air condition of the target environment may be displayed for comparison. This allows the user to more clearly grasp the air condition of the target environment. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0087] <Functional configuration of information processing device> The functional configuration of the information processing device 120 according to the second embodiment will be described. Fig. 19 is a second diagram showing an example of the functional configuration of the information processing device. The difference from the functional configuration described using Fig. 3 in the first embodiment is that comparison environment data is input to the environment data acquisition unit 320 from a comparison environment data storage unit 1910.

[0088] The comparison environment data storage unit 1910 stores the comparison environment data. The comparison environment data includes items similar to the following: each item included in the environmental data notified to the environmental data acquisition unit 320 by the communication control unit 310, each item included in the environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), and each item included in the environmental data input by the user.

[0089] However, the item values ​​of each item in the comparison environment data have item values ​​(e.g., item values ​​in a poor environment) that are different from: - the item values ​​of each item included in the environmental data notified from the communication control unit 310, and - the item values ​​of each item included in the environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), and - the item values ​​of each item included in the environmental data input by the user. Specifically, the item values ​​of each item in the comparison environment data are item values ​​that indicate an air condition opposite to the air condition in the target environment (the air condition in the comparison environment), or item values ​​that affect an air condition opposite to the air condition in the target environment (the air condition in the comparison environment).

[0090] The environmental data acquisition unit 320 acquires environmental data, notifies the acquired environmental data to the environmental language generation unit 330 and the prompt generation unit 340, and reads out comparative environmental data. The environmental data acquisition unit 320 notifies the environmental language generation unit 330 and the prompt generation unit 340 of the read out comparative environmental data.

[0091] The prompt generator 340 generates a prompt for generating an image that sensorily represents the air condition of the comparison environment, in addition to a prompt for generating an image that sensorily represents the air condition of the target environment. This enables the output unit 350 to obtain the image that sensorily represents the air condition of the target environment and the image that sensorily represents the air condition of the comparison environment from the server device 110 via the communication controller 310 and display them in comparison on the display screen.

[0092] <Display Example> A description will be given of a display example in which the information processing device 120 displays an image generated by the server device 110 based on a prompt generated by the information processing device 120. Fig. 20 is a second diagram showing an example of a display screen.

[0093] 20 , display screen 2000 has area 2010 for displaying an image that intuitively expresses the air condition of the target environment, and area 2011 for displaying a sentence included in the prompt used to generate the image. Display screen 2000 also has area 2020 for displaying an image that intuitively expresses the air condition of a comparison environment, and area 2021 for displaying a sentence included in the prompt used to generate the image.

[0094] In this way, by displaying an image that intuitively expresses the air condition of the target environment and an image that intuitively expresses the air condition of the comparison environment, for example, the user can more clearly understand that the air condition of the target environment is the air condition of an ideal environment.

[0095] In the example of Fig. 20, an image that intuitively represents the air condition of a poor environment is displayed as an image that intuitively represents the air condition of a comparison environment. However, if the target environment is a poor environment, an image that intuitively represents the air condition of an ideal environment may be displayed. This makes it possible to provide a user interface that, for example, when a user selects an image that intuitively represents the air condition of the ideal environment, controls the air conditioning to create the ideal environment.

[0096] <Summary> As is clear from the above explanation, the information processing device 120 according to the second embodiment: When displaying an image showing the air condition of a target environment, it displays an image showing the air condition of a comparison environment that is different from the target environment.

[0097] As a result, according to the second embodiment, the user can more clearly grasp the state of the air in the target environment.

[0098] [Third Embodiment] In the first and second embodiments, the information processing device 120 has been described as causing the generation AI 111 to generate a still image. However, the information processing device 120 may also cause the generation AI 111 to generate a moving image. The following describes the third embodiment, focusing on the differences from the first embodiment.

[0099] <Functional configuration of information processing device> The functional configuration of the information processing device 120 according to the third embodiment will be described. Fig. 21 is a third diagram showing an example of the functional configuration of the information processing device. The functional configuration shown in Fig. 21 differs from the functional configuration described using Fig. 3 in the first embodiment in that it includes a motion control language generation unit 2110.

[0100] The control table storage unit 2120 stores a control table. The control table mainly has items similar to the items included in the environmental data notified to the environmental data acquisition unit 320 by the communication control unit 310. The control table also has an action control language that indicates the degree of action of each component in the video, associated with the item value of each item. The component here refers to the component (e.g., clouds, plants, flowers, rivers, etc.) included in the video generated by the generation AI.

[0101] The movement control language generation unit 2110 references the control table stored in the control table storage unit 2120 to generate a movement control language based on the environmental data notified by the environmental data acquisition unit 320. The movement control language generation unit 2110 notifies the prompt generation unit 340 of the generated movement control language.

[0102] The prompt generation unit 340 adds the action control language notified by the action control language generation unit 2110 to the wording or sentence included in the prompt.

[0103] <Specific example of processing by the movement control language generation unit> A specific example of processing by the movement control language generation unit 2110, one of the units of the information processing device 120 according to the third embodiment, will be described. Fig. 22 is a diagram showing a specific example of processing by the movement control language generation unit. As shown in Fig. 22, the control table storage unit 2120 stores, for example, control tables 2210 to 2230.

[0104] Control table 2210 is an example of a table that defines the action control language when the component is "cloud." The example of control table 2210 shows how the action control language is defined for each item value of the item "wind speed" and for each altitude.

[0105] Control table 2220 is an example of a table that defines the action control language when the component is "plants, trees, flowers." The example of control table 2220 shows how the action control language is defined for each item value of the item = "ground wind speed."

[0106] Control table 2230 is an example of a table that defines an action control language when the component is a "river." The example of control table 2230 shows how an action control language is defined for each item value of the item = "rainfall."

[0107] For example, when the environmental data acquisition unit 320 notifies the movement control language generation unit 2110 of the wind speed at a specific altitude as environmental data, the movement control language generation unit 2110 refers to the control table 2210. The movement control language generation unit 2110 generates movement control language that represents the degree of movement (flow) of clouds according to the wind speed at the specific altitude, and notifies the prompt generation unit 340. This allows the prompt generation unit 340 to add the movement control language to the sentence component = "cloud" included in the prompt.

[0108] For example, when the environmental data acquisition unit 320 notifies the movement control language generation unit 2110 of ground wind speed as environmental data, the movement control language generation unit 2110 refers to the control table 2220. The movement control language generation unit 2110 generates movement control language that indicates the degree of movement (swaying) of plants and flowers according to the ground wind speed, and notifies the prompt generation unit 340. This allows the prompt generation unit 340 to add the movement control language to the sentence components of the prompt, "plants, flowers."

[0109] For example, when the environmental data acquisition unit 320 notifies the movement control language generation unit 2110 of the amount of rainfall as environmental data, the movement control language generation unit 2110 refers to the control table 2220. The movement control language generation unit 2110 generates movement control language that indicates the degree of movement (flow) of the river depending on the amount of rainfall, and notifies the prompt generation unit 340. This allows the prompt generation unit 340 to add movement control language when the sentence component included in the prompt contains "river."

[0110] <Examples of prompts and moving images> An example of a prompt generated by the information processing device 120 according to the third embodiment and an example of a moving image generated by the generation AI 111 of the server device 110 using the prompt will be described. Fig. 23 is a diagram showing an example of a prompt (sentence) and an example of a moving image.

[0111] Among these, reference numeral 2310 indicates a generated prompt that includes "clouds" as a component and has "quickly" added as an action control word. Reference numeral 2311 indicates an example of a moving image (a moving image showing clouds moving quickly) generated by the generation AI 111 of the server device 110 based on a prompt that includes the sentence indicated by reference numeral 2310.

[0112] Reference numeral 2320 indicates that the generated prompt includes the components "river," "flower," and "cloud," but no motion control language is added to any of the components. In this case, the generation AI 111 of the server device 110 generates a video in which each component operates based on the default motion control language. Reference numeral 2321 indicates an example of a video (a video showing a river and clouds flowing at their default speeds and flowers swaying at their default speeds) generated by the server device 110 based on a prompt including the sentence indicated by reference numeral 2320.

[0113] Reference numeral 2330 indicates a generated prompt that includes "river" as a component and adds "strongly" as an action control word. Reference numeral 2331 indicates an example of a moving image (a moving image showing a vigorously flowing river) generated by the generation AI 111 of the server device 110 based on a prompt that includes the sentence indicated by reference numeral 2330.

[0114] <Flow of image generation processing by image generation system> A flow of image generation processing by the image generation system 100 including the information processing device 120 according to the third embodiment will be described. Fig. 24 is a second flowchart showing the flow of image generation processing by the image generation system.

[0115] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0116] In step S1702, the information processing apparatus 120 generates an environmental language by converting the acquired environmental data into a natural language including sensory expressions.

[0117] In step S2401, the information processing device 120 generates a motion control language based on the acquired environmental data.

[0118] In step S1703, the information processing device 120 generates a prompt based on either or both of the generated environmental language and the acquired environmental data. The information processing device 120 adds corresponding action control language to the components of the sentence included in the generated prompt. The prompt generated by the information processing device 120 includes: - a phrase or sentence expressing an image co-occurring with either or both of the environmental data and the environmental language (the component has the action control language added); - an image expressing an image co-occurring with either or both of the environmental data and the environmental language; and - image editing parameters according to the environmental data.

[0119] In step S2403, the information processing device 120 transmits the generated prompt to the server device 110. As a result, the generation AI 111 of the server device 110 generates an image (moving image) according to the prompt.

[0120] In step S2404, the information processing device 120 acquires the image (moving image) generated by the generation AI 111 of the server device 110 and displays it to the user.

[0121] In step S1706, the information processing apparatus 120 determines whether or not to end the image generation process. If it is determined in step S1706 that the image generation process should be continued (NO in step S1706), the process returns to step S1701.

[0122] On the other hand, if it is determined in step S1706 that the image generation process is to be ended (YES in step S1706), the image generation process is ended.

[0123] <Summary> As is clear from the above explanation, the information processing device 120 according to the third embodiment performs the following operations in addition to the processes performed by the information processing device 120 according to the first embodiment: Generates a movement control language that indicates the degree of movement of a component based on environmental data. Adds the movement control language to a component of a sentence included in a prompt. Operates the generation AI using the prompt, and displays a moving image that indicates the air condition of the target environment, generated by the generation AI, in which the component moves based on the movement control language.

[0124] As a result, the information processing device 120 according to the third embodiment can generate a moving image that intuitively expresses the state of the air in the target environment.

[0125] [Fourth embodiment] In each of the above embodiments, an image is acquired by operating the generation AI 111 of the server device 110. In contrast, in this embodiment and the following embodiments, an image is acquired by operating the generation AI 111 of the server device 110 in the generation phase after performing additional learning in the learning phase. The fourth embodiment will be described below, focusing on the differences from the above embodiments.

[0126] <System configuration of image generation system> A system configuration in the learning phase of an image generation system to which an information processing device according to the fourth embodiment is applied will be described. Fig. 25 is a second diagram showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the learning phase.

[0127] 25 , an image generation system 2500 includes a server device 110, an information processing device 120, and a server device 2540. In the image generation system 2500, the information processing device 120, the server device 110, and the server device 2540 are communicatively connected via a network 150.

[0128] In the learning phase, the server device 110 has a generation AI 111 and a fine-tuning unit 2511. When the server device 110 receives learning data from the information processing device 120 via the network 150, the server device 110 uses the fine-tuning unit 2511 to perform additional learning on the generation AI 111. As a result, the fine-tuning unit 2511 generates tuned models corresponding to each piece of learning data.

[0129] In the learning phase, the information processing device 120 generates learning data and instructs the generation AI 111 to perform additional learning. Specifically, the information processing device 120 acquires a group of images (an example of a group of specific images) to be included in the learning data from the server device 2540. The information processing device 120 generates learning data including the acquired group of images and captions indicating each component included in each image. The information processing device 120 transmits the generated learning data to the server device 110. The information processing device 120 instructs the generation AI 111 to perform additional learning using a model (in this embodiment, LoRA) for generating each image (an example of each specific image) included in the generated learning data.

[0130] In the learning phase, the information processing device 120 calculates a component-specific score for each piece of learning data based on the score indicating the importance of each component in the image, and stores the score in association with the learning data.

[0131] The server device 2540 functions as an information providing unit 2541 and provides images to be included in the learning data to the information processing device 120 via the network 150 .

[0132] <Functional configuration of information processing device> The functional configuration of the information processing device 120 in the learning phase will be described. Fig. 26 is a fourth diagram showing an example of the functional configuration of the information processing device. An information processing program is installed in the information processing device 120, and by executing the information processing program in the learning phase, the information processing device 120 functions as: a communication control unit 310, an image data collection unit 2610, a caption generation unit 2620, a learning data generation unit 2630, and an additional learning unit 2640.

[0133] The communication control unit 310 acquires images from the server device 2540 via the network 150 and notifies the image data collection unit 2610. The communication control unit 310 acquires learning data and instructions for additional learning from the additional learning unit 2640 and transmits them to the server device 110 via the network 150.

[0134] The image data collection unit 2610 stores the image notified by the communication control unit 310 in the image data storage unit 2650 .

[0135] The caption generation unit 2620 reads images stored in the image data storage unit 2650 and performs image recognition processing on the read images to recognize each component included in the read images and calculate the reliability of each recognized component. The caption generation unit 2620 then generates captions indicating each component included in each image. The caption generation unit 2620 notifies the learning data generation unit 2630 of the read images, the generated captions, and the calculated reliability of each component.

[0136] The learning data generation unit 2630 acquires the images, captions, and reliability of each component notified by the caption generation unit 2620. The learning data generation unit 2630 classifies the images notified by the caption generation unit 2620 into multiple groups according to the type of image content. The learning data generation unit 2630 calculates a score for each component for each group based on the reliability of each component corresponding to each image belonging to the classified group.

[0137] The training data generation unit 2630 generates training data including images belonging to the classified groups and captions corresponding to the images. The training data generation unit 2630 associates the generated training data with the scores for each component element calculated for the group, and stores the data in the training data storage unit 2660.

[0138] The additional learning unit 2640 sequentially reads out the learning data generated for each group and notifies the communication control unit 310 together with an instruction for additional learning. This allows the communication control unit 310 to sequentially transmit the read learning data to the server device 110 and instruct the generation AI 111 to perform additional learning using the generated learning data.

[0139] <Specific Examples of Processing by Each Unit of Information Processing Apparatus> A specific example of processing by each unit of the information processing apparatus 120 (here, the caption generation unit 2620 and the learning data generation unit 2630) will be described.

[0140] (1) Specific Example of Processing by the Caption Generation Unit Fig. 27 is a diagram showing a specific example of processing by the caption generation unit. As described above, the caption generation unit 2620 reads an image stored in the image data storage unit 2650 and performs image recognition processing on the read image to recognize each component included in the read image and calculate the reliability of each recognized component.

[0141] In Figure 27, reference numeral 2710 indicates an image read by the caption generation unit 2620 from the image data storage unit 2650. In Figure 27, reference numeral 2720 indicates the recognition of each component, such as "sunsetcloud", "scenery", "sun", "sky", "cloud", and "mountain", by performing image recognition processing on the read image by the caption generation unit 2620. Also in Figure 27, reference numeral 2720 indicates the calculation of the reliability of each component when the caption generation unit 2620 performs image recognition processing on the read image. The caption generation unit 2620 notifies the learning data generation unit 2630 of the read image, captions indicating each component included in the image, and the calculated reliability of each component.

[0142] (2) Specific Example of Processing by the Training Data Generation Unit Figure 28 is a diagram showing a specific example of processing by the training data generation unit. As described above, the training data generation unit 2630 acquires images, captions indicating each component included in the image, and the reliability of each component from the caption generation unit 2620, and classifies the acquired images into multiple groups according to the type of image content. The training data generation unit 2630 calculates a score for each component for each group based on the reliability of each component corresponding to each image belonging to the classified group.

[0143] 28 , reference numeral 2710 denotes an image notified by the caption generation unit 2620, and reference numeral 2720 denotes a caption notified by the caption generation unit 2620. Reference numeral 2730 denotes the reliability of each component element notified by the caption generation unit 2620.

[0144] In Figure 28, reference numeral 2800 indicates that the learning data generation unit 2630 has generated learning data (for sunsets) by: extracting images with image content type = "sunset" from the images shown by reference numeral 2710 and classifying them into a group with image content type = "sunset" (see reference numeral 2810), and associating captions corresponding to images with image content type = "sunset" with images with image content type = "sunset" (see reference numeral 2820).

[0145] 28 , reference numeral 2830 indicates that the learning data generation unit 2630: reads out the reliability of each component element corresponding to images belonging to the group with image content type = "sunset" from the reliability of each component element indicated by reference numeral 2730, and adds up the reliability for each component element and divides by the number of images belonging to the group with image content type = "sunset." It should be noted that when associating the component-specific scores with the learning data (for sunset), components with scores below a predetermined threshold may be excluded.

[0146] <Details of Server Apparatus> The following describes details of the generation AI 111 and the fine tuning unit 2511 of the server apparatus 110 in the fourth embodiment. Fig. 29 is a diagram showing details of the generation AI and the fine tuning unit.

[0147] As shown in Figure 29, the generation AI 111 has a VAE (Variational Auto-Encoder) 2911, an encoder 2912, a decoder 2913, and a VAE 2914. The fine-tuning unit 2511 has a LoRA (Low-Rank Adaptation) 2920. The LoRA 2920 has, for example, a sunset LoRA 2921 used when performing additional training on the generation AI 111 using training data (for sunsets), a sardine cloud LoRA 2922 used when performing additional training on the generation AI 111 using training data (for sardine clouds), and a landscape LoRA 2923 used when performing additional training on the generation AI using training data (for landscapes).

[0148] For example, when additional learning is performed on the generation AI using the LoRA 2921 for sunset, the image shown by reference numeral 2810 of the learning data (for sunset) shown by reference numeral 2800 in Figure 28 and the caption shown by reference numeral 2820 are input to the generation AI 111. The model parameters of the generation AI 111 are fine-tuned so that the image reproduced by the generation AI 111 matches the image shown by reference numeral 2810 (the image input to the generation AI 111).

[0149] <Flow of Additional Learning Process by Image Generation System> A description will be given of the flow of additional learning process in the learning phase by the image generation system 2500. Fig. 30 is a flowchart showing the flow of additional learning process by the image generation system.

[0150] In step S3001, the information processing device 120 collects images provided from the server device 2540.

[0151] In step S3002, the information processing device 120 performs image recognition processing on the collected image to recognize each component included in the image and calculate the reliability of each recognized component. The information processing device 120 generates a caption indicating each component included in the image.

[0152] In step S3003, the information processing device 120 generates groups of additional learning units by classifying the collected images into a plurality of groups according to the type of image content.

[0153] In step S3004, the information processing apparatus 120 calculates a component-specific score for each group based on the reliability of each component.

[0154] In step S3005, the information processing apparatus 120 generates learning data in which images and captions are associated with each other for each group of additional learning units.

[0155] In step S3006, the information processing device 120 sequentially transmits the learning data generated for each group of additional learning units to the server device 110, thereby performing additional learning on the generated AI 111 using each LoRA possessed by the fine-tuning unit 2511.

[0156] In step S3007, the information processing apparatus 120 determines whether to end the additional learning process. If it is determined in step S3007 that the additional learning process is to be continued (NO in step S3007), the process returns to step S3001.

[0157] On the other hand, if it is determined in step S3007 that the additional learning process is to be ended (YES in step S3007), the additional learning process is ended.

[0158] <System configuration of image generation system> A system configuration in the generation phase of an image generation system to which an information processing device according to the fourth embodiment is applied will be described. Fig. 31 is a third diagram showing an example of the system configuration of the image generation system, and is a diagram showing the system configuration in the generation phase.

[0159] 31 , an image generation system 3100 includes a server device 110, an information processing device 120, and a server device 140. In the image generation system 3100, the information processing device 120, the server device 110, and the server device 140 are communicatively connected via a network 150.

[0160] In the generation phase, the server device 110 has a generation AI 111 and a tuned model 3110. In the generation phase, when the server device 110 receives a prompt from the information processing device 120 via the network 150, it operates the generation AI 111 using the tuned model 3110 to generate an image according to the prompt. The server device 110 transmits the generated image to the information processing device 120 via the network 150.

[0161] 1 in the first embodiment, the information processing device 120 will not be described here. However, in the case of the information processing device 120 according to the fourth embodiment, the prompt sent to the server device 110 includes: a designation of the tuned model 3110 (execution LoRA) to be used; and weights to be used when the tuned model is used (weights according to the type of image content of each of the multiple learning data sets (weights for each execution LoRA)).

[0162] The server device 140 is the same as the server device 140 described in the first embodiment with reference to FIG. 1, and therefore a description thereof will be omitted here.

[0163] <Functional configuration of information processing device> A functional configuration in the generation phase of the information processing device 120 according to the fourth embodiment will be described. Fig. 32 is a fifth diagram showing an example of the functional configuration of the information processing device. The difference from the functional configuration described using Fig. 3 in the first embodiment is that a weight calculation unit 3210 is included.

[0164] The weight calculation unit 3210 acquires the wording or sentences included in the prompt generated by the prompt generation unit 340 and extracts the components included in each training data. The weight calculation unit 3210 reads the component-specific scores associated with the training data stored in the training data storage unit 2660, and extracts the components corresponding to the read component-specific scores from the prompt.

[0165] The weight calculation unit 3210 calculates the sum (total score) of the scores of each component element for each learning data item from the scores of each learning data item corresponding to the component elements extracted from the prompt.

[0166] The weight calculation unit 3210 compares the total scores of each piece of training data and selects the training data with the highest total score to designate the corresponding LoRA as the execution LoRA. The weight calculation unit 3210 calculates a weight for each execution LoRA based on the total score of each piece of selected training data.

[0167] The weight calculation unit 3210 notifies the prompt generation unit 340 of the weight of each execution LoRA. This allows the prompt generation unit 340 to generate a prompt that includes the designation of the execution LoRA and the weight of each execution LoRA.

[0168] As a result, according to the information processing device 120 of the fourth embodiment, it is possible to select a generated AI that has undergone additional training in advance using each of a plurality of pieces of training data based on the generated prompt, and to operate the AI ​​based on the assigned weights. Note that "selecting a generated AI that has undergone additional training in advance" here refers to, for example, selecting a combination of execution LoRAs in a generated AI that includes a plurality of LoRAs that have been generated by additional training in advance.

[0169] <Specific Example of Processing by Weight Calculation Unit> A specific example of processing by the weight calculation unit 3210 will be described below. Fig. 33 is a first diagram showing a specific example of processing by the weight calculation unit.

[0170] In FIG. 33, reference numeral 3310 denotes an example of a sentence included in a prompt generated by the prompt generating unit 340 .

[0171] The weight calculation unit 3210 acquires the sentence (reference numeral 3310) included in the prompt and extracts from the acquired sentence components that correspond to the component-specific scores (reference numerals 3331 to 3333) associated with each learning data. In the sentence (reference numeral 3310) included in the prompt, the underlined components indicate the components extracted by the weight calculation unit 3210 as the components that correspond to the component-specific scores (reference numerals 3331 to 3333) associated with each learning data.

[0172] The weight calculation unit 3210 calculates the sum (total score) of the scores of each component of each training data set from the scores of each training data set corresponding to the component extracted from the prompt (underlined symbol 3310). In Figure 33, symbol 3321 indicates that: a total score of "399" was calculated from the scores of the components extracted from the prompt (underlined symbol 3310) among the components corresponding to the component-specific scores (symbol 3331) associated with the training data (for sardine clouds); a total score of "376" was calculated from the scores of the components extracted from the prompt (underlined symbol 3310) among the components corresponding to the component-specific scores (symbol 3332) associated with the training data (for sunsets); and a total score of "347" was calculated from the scores of the components extracted from the prompt (underlined symbol 3310) among the components corresponding to the component-specific scores (symbol 3333) associated with the training data (for landscape paintings).

[0173] The weight calculation unit 3210 compares the total scores of each piece of training data and selects the training data with the highest total score, thereby designating the corresponding LoRA as the execution LoRA. In the example of Figure 33, due to space limitations, the total scores of three pieces of training data are calculated, and the three pieces of training data are selected, thereby designating the mackerel cloud LoRA, the sunset LoRA, and the landscape LoRA as the execution LoRA (see reference numeral 3322).

[0174] The weight calculation unit 3210 calculates a weight for each LoRA based on the total score of each selected training data. In Fig. 33, reference numeral 3323 indicates that the weights of each execution LoRA corresponding to each training data are calculated from the ratio of the total scores of the training data (sardine cloud), training data (sunset), and training data (landscape) as follows: 0.34, 0.36, 0.30.

[0175] The weight calculation unit 3210 notifies the prompt generation unit 340 of the weight of each execution LoRA. Reference numeral 3324 indicates that the prompt generation unit 340 has been notified of the following: Weight of mackerel cloud LoRA = 0.34 Weight of sunset LoRA = 0.36 Weight of landscape LoRA = 0.30

[0176] <Flow of Image Generation Processing by Image Generation System> The flow of image generation processing in the generation phase by the image generation system 3100 will be described. Fig. 34 is a third flowchart showing the flow of image generation processing by the image generation system. Note that the difference from the first flowchart shown in Fig. 17 is step S3401.

[0177] In step S3401, the information processing device 120 extracts components corresponding to the component-specific scores associated with each training data from the sentence included in the generated prompt. The information processing device 120 calculates the sum (total score) of the scores for each component for each training data from the scores corresponding to the extracted components. The information processing device 120 designates an execution LoRA based on the total score for each training data and calculates a weight for each execution LoRA. The information processing device 120 includes the designation of the execution LoRA and the weight for each execution LoRA in the prompt.

[0178] <Summary> As is clear from the above explanation, in the learning phase, the information processing device 120 according to the fourth embodiment: recognizes each component included in the collected images and calculates the reliability of each recognized component; classifies the collected images into multiple groups according to the type of image content, and calculates component-specific scores for the components included in the images belonging to each group; generates learning data for each group in which images are associated with captions indicating each component included in the image, and associates component-specific scores with each piece of learning data; and generates a tuned model by using the generated learning data and the corresponding LoRA to perform additional learning on the generation AI.

[0179] Furthermore, in the generation phase, the information processing device 120 according to the fourth embodiment: - Generates environmental language by acquiring environmental data of the target environment and converting it into natural language including sensory expressions. - Generates prompts including phrases, sentences, or images that express images that co-occur from the environmental language or environmental data. - Specifies an execution LoRA based on the components of the sentence included in the generated prompt, and calculates the weight of each execution LoRA. - Operates the generation AI using the execution LoRA by using a prompt including the designation of the execution LoRA and the weight of each execution LoRA. Also, by operating the generation AI, an image showing the air condition of the target environment, generated by the generation AI, is acquired and displayed to the user.

[0180] In this way, by operating a generation AI that has undergone additional learning using learning data generated for each type of image content, the information processing device 120 according to the fourth embodiment can generate images that intuitively express the state of the air with high accuracy.

[0181] [Fifth Embodiment] In the fourth embodiment, the weights of the execution LoRA are calculated by referring to the component-specific scores associated with the learning data. However, the calculation method of the weights of the execution LoRA is not limited to this, and the weights of the execution LoRA may be calculated by other calculation methods. The following describes the fifth embodiment, focusing on the differences from the fourth embodiment.

[0182] <Specific example of processing by weight calculation unit> A specific example of processing by the weight calculation unit 3210 of the information processing device 120 according to the fifth embodiment will be described. Fig. 35 is a second diagram showing a specific example of processing by the weight calculation unit. Differences from the first diagram shown in Fig. 33 are reference numerals 3421 and 3422.

[0183] As indicated by the reference numeral 3421, in the information processing device 120 according to the fifth embodiment, the weight calculation unit 3210 uses the LLM to determine the importance of the word represented by the execution LoRA in the sentence included in the prompt. Note that LLM is an abbreviation for Large Language Model, and refers to a large-scale language model.

[0184] The example of Figure 35 shows a case where the execution LoRAs are specified as a LoRA for sardine clouds, a LoRA for sunsets, and a LoRA for landscapes, and also shows a case where the importance of the words represented by the specified execution LoRAs, "sardine clouds," "sunsets," and "landscapes," in the sentence (reference numeral 3310) included in the prompt is determined.

[0185] According to the LLM, the "sunset" is "an element that determines the overall atmosphere, and influences the other elements with light and color," so its importance is judged to be "50%." According to the LLM, the "sardine clouds" are "beautifully illuminated by the light of the sunset, and their shadows add depth to the landscape," so their importance is judged to be "30%." According to the LLM, the "landscape painting" is "further enhanced by the sunset and sardine clouds, creating overall beauty, but does not play as central a role as the sunset and sardine clouds," so its importance is judged to be "20%."

[0186] As a result, the weight calculation unit 3210 calculates the weights of the respective execution LoRAs as 0.5, 0.3, and 0.2.

[0187] In the information processing device 120 according to the fifth embodiment, the weight calculation unit 3210 notifies the prompt generation unit 340 of the weight of each execution LoRA. Reference numeral 3422 indicates that the prompt generation unit 340 has been notified of the following: Weight of mackerel cloud LoRA = 0.5 Weight of sunset LoRA = 0.3 Weight of landscape LoRA = 0.2

[0188] <Summary> As is clear from the above explanation, in the generation phase, the information processing device 120 according to the fifth embodiment: - Generates environmental language by acquiring environmental data of the target environment and converting it into natural language including sensory expressions. - Generates prompts including phrases, sentences, or images that express images that co-occur from the environmental language or environmental data. - Specifies an execution LoRA based on the components of the sentence included in the generated prompt. - Calculates the weight of each execution LoRA by determining the importance of the words represented by each execution LoRA in the sentence included in the prompt using LLM. - Displays an image indicating the air condition of the target environment, generated by the generation AI, by operating the generation AI using a prompt including the specification of the execution LoRA and the weight of each execution LoRA.

[0189] In this way, by operating a generation AI that has undergone additional learning using learning data generated for each type of image content, the information processing device 120 of the fifth embodiment can generate images that intuitively express the state of the air with high accuracy, as in the fourth embodiment.

[0190] [Sixth Embodiment] In the fourth and fifth embodiments, the execution LoRA is designated based on the sentence included in the generated prompt. However, the method of designating the execution LoRA is not limited to this. For example, conditions for designating the execution LoRA (referred to as environmental conditions) may be set, and the execution LoRA may be designated when the environmental conditions are met. The sixth embodiment will be described below, focusing on the differences from the fourth and fifth embodiments.

[0191] <Functional configuration of information processing device> The functional configuration of the information processing device 120 in the generation phase according to the sixth embodiment will be described. Fig. 36 is a sixth diagram showing an example of the functional configuration of the information processing device. The difference from the functional configuration described using Fig. 32 in the fourth embodiment is that a selection unit 3610 is provided instead of the weight calculation unit 3210.

[0192] The selection unit 3610 acquires the wording or sentences included in the prompt generated by the prompt generation unit 340 and extracts the components included in each training data. The selection unit 3610 reads the component-specific scores associated with the training data stored in the training data storage unit 2660, and extracts the components corresponding to the read component-specific scores from the prompt.

[0193] The selection unit 3610 calculates the sum (total score) of the scores of each component element for each learning data item from the scores of each learning data item corresponding to the component elements extracted from the prompt.

[0194] The selection unit 3610 compares the total scores of each training data and selects the training data with the highest total score to designate the corresponding LoRA as the execution LoRA. At this time, the selection unit 3610 determines whether or not environmental conditions are assigned to the selected training data, and if environmental conditions are assigned, determines whether or not the sentences included in the generated prompt are consistent with the environmental conditions.

[0195] If the selection unit 3610 determines that environmental conditions are assigned to the selected learning data and that the sentences contained in the generated prompt are consistent with the environmental conditions, it does not change the specified execution LoRA.

[0196] On the other hand, if it is determined that the selected learning data has environmental conditions attached to it and that the sentence contained in the generated prompt does not match the environmental conditions, the selection unit 3610 re-designates the LoRA corresponding to other learning data as the execution LoRA.

[0197] Alternatively, if it is determined that the selected learning data does not have an environmental condition, the selection unit 3610 re-designates the LoRA corresponding to any other learning data as the execution LoRA.

[0198] The selector 3610 equally distributes weights to each specified execution LoRA or each re-specified execution LoRA, and notifies the prompt generator 340 of the weights equally distributed to each execution LoRA.

[0199] <Specific Example of Processing by Selector> A specific example of processing by the selector 3610 will be described below. Fig. 37 is a diagram showing a specific example of processing by the selector.

[0200] In FIG. 37, reference numeral 3310 is the same as reference numeral 3310 shown in FIG. 33, and reference numerals 3321 and 3322 are the same as reference numerals 3321 and 3322 shown in FIG. 33, and therefore, description thereof will be omitted here.

[0201] As indicated by reference numeral 3721, the selection unit 3610 determines whether or not environmental conditions have been assigned to the selected training data. Here, it is assumed that the selection unit 3610 has selected training data (for sardine clouds), training data (for sunsets), and training data (for landscapes). Furthermore, as indicated by reference numerals 3731 to 3733, it is assumed that: · the training data (for sardine clouds) is not assigned environmental conditions; · the training data (for sunsets) is assigned environmental conditions of time = evening and weather = sunny; · the training data (for landscapes) is assigned environmental conditions of location = anywhere.

[0202] In this case, the selection unit 3610: determines that environmental conditions have been assigned to the learning data (for sunsets) and that the sentence included in the prompt satisfies the assigned environmental conditions, and does not change the designation of the corresponding sunset LoRA as the LoRA to be executed; determines that environmental conditions have been assigned to the learning data (for landscapes) and that the sentence included in the prompt satisfies the assigned environmental conditions, and does not change the designation of the corresponding landscape LoRA as the LoRA to be executed; and, because environmental conditions have not been assigned to the learning data (for mackerel clouds), re-designates any other LoRA, the temperature LoRA, as the LoRA to be executed (see reference numeral 3721).

[0203] As indicated by reference numeral 3722, the selection unit 3610 equally distributes weights (for example, 0.3 each) to each execution LoRA (here, the sunset LoRA, the landscape LoRA, and the temperature LoRA).

[0204] As shown by the reference numeral 3723, the selection unit 3610 notifies the prompt generation unit 340 of the weights (examples of predetermined weights) of each specified execution LoRA or each re-specified execution LoRA: - Weight of sunset LoRA = 0.3, - Weight of landscape LoRA = 0.3, - Weight of temperature LoRA = 0.3.

[0205] <Summary> As is clear from the above description, in the generation phase, the information processing device 120 according to the sixth embodiment: - Generates environmental language by acquiring environmental data of the target environment and converting it into natural language including sensory expressions. - Generates prompts including phrases, sentences, or images expressing images co-occurring from the environmental language or environmental data. - Specifies an execution LoRA based on the components of the sentence included in the generated prompt. - Determines whether the sentence included in the generated prompt satisfies the environmental conditions assigned to the learning data, and if it is determined that the sentences do not satisfy the environmental conditions, re-specifies the execution LoRA. - Evenly distributes weights to each execution LoRA. - Operates the generation AI using the execution LoRA by using a prompt including the designation of the execution LoRA and the weight of each execution LoRA. Furthermore, by operating the generation AI, an image indicating the air condition of the target environment generated by the generation AI is obtained and displayed to the user.

[0206] In this way, the information processing device 120 according to the sixth embodiment operates the generation AI that has undergone additional learning using the learning data generated for each type of image content, taking into account the environmental conditions assigned to the learning data and based on a predetermined weight. As a result, the information processing device 120 according to the sixth embodiment can appropriately generate an image that intuitively expresses the state of the air.

[0207] [Seventh Embodiment] In the fourth embodiment, the weight calculation unit 3210 calculates the weight of each execution LoRA based on the total score. In contrast, in the seventh embodiment, multiple combinations of weights for each execution LoRA are prepared, and the generation AI is operated using each execution LoRA under each combination, and the user evaluates each generated image. This makes it possible to generate images under an appropriate combination of weights. The seventh embodiment will be described below, focusing on the differences from the fourth embodiment.

[0208] <Functional Configuration of Information Processing Device> The functional configuration of the information processing device 120 in the generation phase according to the seventh embodiment will be described. Fig. 38 is a seventh diagram showing an example of the functional configuration of the information processing device. The differences from the functional configuration described using Fig. 32 in the above fourth embodiment are that the function of the weight calculation unit 3810 is different from the function of the weight calculation unit 3210 in Fig. 32 and that the information processing device 120 according to the seventh embodiment has a weight storage unit 3820.

[0209] The weight calculation unit 3810 obtains the wording or sentences included in the prompt generated by the prompt generation unit 340 and extracts the components included in each training data. The weight calculation unit 3810 reads the component-specific scores associated with the training data stored in the training data storage unit 2660, and extracts the components corresponding to the read component-specific scores from the prompt.

[0210] The weight calculation unit 3810 calculates the sum (total score) of the scores of each component element for each learning data item from the scores of each learning data item corresponding to the component elements extracted from the prompt.

[0211] The weight calculation unit 3810 compares the total scores of each piece of training data and selects the training data with the highest total score to designate the corresponding LoRA as the execution LoRA. The weight calculation unit 3810 notifies the prompt generation unit 340 of multiple combinations of weights for each designated execution LoRA.

[0212] This allows the generation AI 111 to operate using each execution LoRA under multiple weight combinations. Each image corresponding to each combination generated by the generation AI 111 can be displayed to the user.

[0213] As a result, the user evaluates each image, and the weight calculation unit 3810 receives the evaluation results from the user. The weight calculation unit 3810 stores in the weight storage unit 3820 the combination of weights that has been most highly evaluated by the user from among the received evaluation results.

[0214] <Specific example of processing by weight calculation unit> A specific example of processing by the weight calculation unit 3810 of the information processing device 120 according to the seventh embodiment will be described. Fig. 39 is a third diagram showing a specific example of processing by the weight calculation unit. Differences from the first diagram shown in Fig. 33 are reference numerals 3921, 3922, 3940, the weight calculation unit 3810, and the weight storage unit 3820.

[0215] As indicated by reference numeral 3921, in the information processing device 120 according to the seventh embodiment, the weight calculation unit 3810 notifies the prompt generation unit 340 of multiple combinations of weights for multiple specified execution LoRAs. In response to notifying the prompt generation unit 340 of the multiple combinations, corresponding images are displayed to the user, and the user's evaluation results are input. As a result, as indicated by reference numeral 3922, the weight calculation unit 3810 stores the weight combination with the highest evaluation among the received evaluation results in the weight storage unit 3820.

[0216] In this way, by storing the highly evaluated weight combination for each of the specified combinations of execution LoRAs in the weight storage unit 3820, it becomes possible to use the combination of execution LoRAs with appropriate weights thereafter.

[0217] 39, the weight calculation unit 3810 is configured to prepare a plurality of predetermined weight combinations and store the weight combination that is most highly rated by the user, thereby optimizing the weight combination. However, the method for optimizing the weight combination is not limited to this. For example, each time the weight combination is changed, the evaluation result for the image based on the weight combination before the change and the evaluation result for the image based on the weight combination after the change may be compared, and the weight combination may be changed in a direction that improves the evaluation result.

[0218] <Flow of Image Generation Processing by Image Generation System> A description will be given of the flow of image generation processing in the generation phase by the image generation system 3100. Fig. 40 is a fourth flowchart showing the flow of image generation processing by the image generation system.

[0219] In step S1701, the information processing device 120 acquires environmental data measured by sensors 1 to n (reference numerals 130_1 to 130_n), environmental data input by the user, and environmental data provided by the server device 140.

[0220] In step S1702, the information processing apparatus 120 generates an environmental language by converting the acquired environmental data into a natural language including sensory expressions.

[0221] In step S1703, the information processing apparatus 120 generates a prompt based on either or both of the generated environment language and the acquired environment data.

[0222] In step S4001, the information processing device 120 extracts components corresponding to the component-specific scores associated with each training data from the sentence included in the generated prompt. The information processing device 120 calculates the sum (total score) of the scores of each component for each training data from the scores corresponding to the extracted components, and specifies the execution LoRA.

[0223] In step S4002, the information processing device 120 determines whether the specified combination of weights for the execution LoRA is stored in the weight storage unit 3820. If it is determined in step S4002 that the combination is stored (YES in step S4002), the process proceeds to step S4003.

[0224] In step S4003, the information processing device 120 sends a prompt including a designation of an execution LoRA and a weight combination of the designated execution LoRA to the server device 110, and operates the generation AI 111 using the execution LoRA under the weight combination, thereby causing the generation AI 111 of the server device 110 to generate an image corresponding to the weight combination.

[0225] In step S1705, the information processing device 120 acquires the image generated by the generation AI 111 and displays it to the user.

[0226] On the other hand, if it is determined in step S4002 that the information is not stored (NO in step S4002), the process proceeds to step S4004.

[0227] In step S4004, the information processing device 120 acquires multiple combinations of weights for the specified execution LoRA.

[0228] In step S4005, the information processing device 120 sequentially transmits the specified combinations of weights of the execution LoRA to the server device 110, including them in prompts, and sequentially operates the generation AI using the execution LoRA under each combination. As a result, the generation AI of the server device 110 generates an image according to each combination.

[0229] In step S4006, the information processing device 120 acquires images corresponding to each combination generated by the generation AI 111 and displays them to the user, allowing the user to evaluate each image.

[0230] In step S4007, the information processing device 120 stores the combination of weights corresponding to the image with the highest evaluation result input by the user in the weight storage unit 3820. The information processing device 120 also displays the image with the highest evaluation result input by the user on the display screen.

[0231] In step S1706, the information processing apparatus 120 determines whether or not to end the image generation process. If it is determined in step S1706 that the image generation process is to be continued (NO in step S1706), the process returns to step S1701.

[0232] On the other hand, if it is determined in step S1706 that the image generation process is to be ended (YES in step S1706), the image generation process is ended.

[0233] <Summary> As is clear from the above explanation, the information processing device 120 according to the seventh embodiment: - Generates environmental language by acquiring environmental data of the target environment and converting it into natural language including sensory expressions. - Generates prompts including words, sentences, or images that express images that co-occur from the environmental language or environmental data. - Specifies an execution LoRA based on the components of the sentence included in the generated prompt. - Operates the generation AI by sequentially including multiple combinations of weights for the specified execution LoRA in the prompt. - A user evaluates multiple images generated by the generation AI, and the weight combination corresponding to the image with the highest evaluation is stored in the weight storage unit.

[0234] In this way, the information processing device 120 according to the seventh embodiment uses a weighting combination that is highly rated by users when operating the generation AI that has undergone additional learning using learning data generated for each type of image content. As a result, the information processing device 120 according to the seventh embodiment can generate images that intuitively represent the state of air with higher accuracy.

[0235] Eighth Embodiment In the fourth to seventh embodiments, the tuned model 3110 of the server device 110 has been described as having a plurality of arbitrary LoRAs. In contrast, in the eighth embodiment, the plurality of LoRAs are categorized into configuration LoRAs, element LoRAs, and environment LoRAs, and the generation AI is operated sequentially using the LoRAs of each category. The following describes the eighth embodiment, focusing on the differences from the fourth to seventh embodiments.

[0236] <System configuration of image generation system> A system configuration in the learning phase of an image generation system to which an information processing device according to the eighth embodiment is applied will be described. Fig. 41 is a fourth diagram showing an example of the system configuration of an image generation system, and is a diagram showing the system configuration in the learning phase.

[0237] 41 , the image generation system 4100 includes a server device 110, an information processing device 120, and a server device 2540. In the image generation system 4100, the information processing device 120, the server device 110, and the server device 2540 are communicatively connected via a network 150.

[0238] In the learning phase, the server device 110 has a generation AI 111, a configuration fine-tuning unit 4111, an element fine-tuning unit 4112, and an environment fine-tuning unit 4113. When the server device 110 receives configuration learning data from the information processing device 120 via the network 150, it uses the configuration fine-tuning unit 4111 to perform additional learning on the generation AI 111. As a result, the configuration fine-tuning unit 4111 generates a configuration-tuned model corresponding to each configuration learning data.

[0239] Similarly, when the server device 110 receives element training data from the information processing device 120 via the network 150, it uses the element fine-tuning unit 4112 to perform additional training on the generation AI 111. As a result, the element fine-tuning unit 4112 generates an element-tuned model corresponding to each element training data.

[0240] Similarly, when the server device 110 receives environmental learning data from the information processing device 120 via the network 150, it uses the environmental fine-tuning unit 4113 to perform additional learning on the generation AI 111. As a result, the environmental fine-tuning unit 4113 generates an environmentally tuned model corresponding to each environmental learning data.

[0241] In the learning phase, the information processing device 120 generates learning data and instructs the generation AI 111 to perform additional learning. Specifically, the information processing device 120 acquires images to be included in the configuration learning data from the server device 2540. The information processing device 120 generates configuration learning data that includes the acquired images and captions indicating each component included in the acquired images. The information processing device 120 transmits the generated configuration learning data to the server device 110 and instructs the generation AI 111 to perform additional learning using the generated configuration learning data.

[0242] Similarly, the information processing device 120 acquires images to be included in the element training data from the server device 2540. The information processing device 120 generates element training data including the acquired images and captions indicating each component included in the acquired images. The information processing device 120 transmits the generated element training data to the server device 110 and instructs the server device 110 to perform additional training on the generation AI 111 using the generated element training data.

[0243] Similarly, the information processing device 120 acquires images to be included in the environmental learning data from the server device 2540. The information processing device 120 generates environmental learning data including the acquired images and captions indicating each component included in the acquired images. The information processing device 120 transmits the generated environmental learning data to the server device 110 and instructs the server device 110 to perform additional learning on the generation AI 111 using the generated environmental learning data.

[0244] During the learning phase, the information processing device 120 calculates a component-specific score for each piece of learning data (each piece of configuration learning data, each piece of element learning data, and each piece of environment learning data) based on a score indicating the importance of each component within the image, and stores the score in association with the learning data.

[0245] The server device 2540 functions as an information providing unit 2541 and provides the images to be included in the configuration learning data, element learning data, and environment learning data to the information processing device 120 via the network 150 .

[0246] <Specific Examples of Learning Data> Specific examples of learning data for each category (here, configuration learning data, element learning data, and environment learning data) generated by the information processing device 120 in the learning phase will be described. Fig. 42 is a diagram showing specific examples of learning data for each category.

[0247] As shown in FIG. 42, the configuration learning data 4210 includes learning data (for sunsets), learning data (for mountains), learning data (for beaches), learning data (for parks), and the like.

[0248] The element training data 4220 includes training data (for clouds), training data (for trees), training data (for flowers), training data (for rivers), and the like.

[0249] The environmental learning data 4230 includes learning data (for temperature), learning data (for humidity), learning data (for wind speed), learning data (for rainfall), and the like.

[0250] The items included in each training data are the same regardless of the type of training data, and include "image data" and "caption" as described with reference to Fig. 28. Furthermore, each training data is associated with a "score by component element."

[0251] <System configuration of image generation system> A system configuration in the generation phase of an image generation system to which an information processing device according to the eighth embodiment is applied will be described. Fig. 43 is a fifth diagram showing an example of the system configuration of an image generation system, and is a diagram showing the system configuration in the generation phase.

[0252] 43 , an image generation system 4300 includes a server device 110, an information processing device 120, and a server device 140. In the image generation system 4300, the information processing device 120, the server device 110, and the server device 140 are communicatively connected via a network 150.

[0253] In the generation phase, the server device 110 has a generation AI 111 and tuned models for each category (a configuration-tuned model 4311, an element-tuned model 4312, and an environment-tuned model 4313). When the server device 110 receives a prompt from the information processing device 120 via the network 150, it operates the generation AI 111 using the configuration-tuned model 4311. As a result, the server device 110 generates a first image as an image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0254] When the server device 110 receives a prompt including a first image from the information processing device 120 via the network 150, it operates the generation AI 111 using the element-tuned model 4312. As a result, the server device 110 generates a second image as an image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0255] When the server device 110 receives a prompt including the second image from the information processing device 120 via the network 150, it operates the generation AI 111 using the environment-tuned model 4313. As a result, the server device 110 generates a third image as an image corresponding to the prompt and transmits it to the information processing device 120 via the network 150.

[0256] The information processing device 120 is similar to the information processing device 120 described in the fourth embodiment using Fig. 31 , and therefore description thereof will be omitted here. However, in the case of the information processing device 120 according to the eighth embodiment, when sending a prompt to the server device 110, the following are sequentially sent: a prompt instructing the server device 110 to generate a first image by operating the generation AI 111 using the configuration-tuned model 4311; a prompt including the first image, instructing the server device 110 to generate a second image by operating the generation AI 111 using the element-tuned model 4312; and a prompt including the second image, instructing the server device 110 to generate a third image by operating the generation AI 111 using the environment-tuned model.

[0257] <Flow of Image Generation Processing by Image Generation System> The flow of image generation processing in the generation phase by the image generation system 4300 will be described. Figure 44 is a fifth flowchart showing the flow of image generation processing by the image generation system. Note that the differences from the first flowchart shown in Figure 17 are steps S4401 to S4404.

[0258] In step S4401, the information processing device 120 extracts components corresponding to the component-specific scores associated with each training data from the sentence included in the generated prompt. The information processing device 120 calculates the sum of the scores of each component for each training data (total score) from the scores corresponding to the extracted components. As a result, based on the total score of each training data, the information processing device 120: specifies the execution LoRA of the configuration-tuned model 4311; specifies the execution LoRA of the element-tuned model 4312; and specifies the execution LoRA of the environment-tuned model 4313.

[0259] In step S4402, the information processing device 120 sends a prompt including a specification of the execution LoRA of the configuration-tuned model 4311 to the server device 110. As a result, the server device 110 operates the generation AI using the execution LoRA of the configuration-tuned model 4311 to generate the first image.

[0260] In step S4403, the information processing device 120 sends to the server device 110 a prompt including the generated first image and a specification of the execution LoRA of the element-tuned model 4312. As a result, the server device 110 operates the generation AI using the execution LoRA of the element-tuned model 4312 to generate the second image.

[0261] In step S4404, the information processing device 120 transmits to the server device 110 a prompt including the generated second image and a specification of the execution LoRA of the environmentally tuned model 4313. As a result, the server device 110 operates the generation AI using the execution LoRA of the environmentally tuned model 4313 to generate the third image.

[0262] <Summary> As is clear from the above description, in the learning phase, the information processing device 120 according to the eighth embodiment: recognizes each component included in the collected images and calculates the reliability of each recognized component; classifies the collected images into multiple groups according to the type of image content, and calculates a component-specific score for each component included in the images belonging to each group; generates training data for each group in which images are associated with captions indicating each component included in the images, and associates a component-specific score for each training data; categorizes the generated training data into composition training data, element training data, and environment training data; uses the composition training data to perform additional training on the generative AI using the corresponding LoRA of the composition fine-tuning unit, thereby generating a composition-tuned model; uses the element training data to perform additional training on the generative AI using the corresponding LoRA of the element fine-tuning unit, thereby generating an element-tuned model; uses the environment training data to perform additional training on the generative AI using the corresponding LoRA of the environment fine-tuning unit, thereby generating an environment-tuned model.

[0263] Furthermore, in the generation phase, the information processing device 120 according to the eighth embodiment: - generates environmental language by acquiring environmental data of the target environment and converting it into natural language including sensory expressions; - generates a prompt including a phrase, sentence, or image expressing an image co-occurring from the environmental language or environmental data; - specifies an execution LoRA for each category based on the components of the sentence included in the generated prompt; - transmits a prompt including a specification of the execution LoRA of the configuration-tuned model. This causes the generation AI to operate using the LoRA of the configuration-tuned model and acquire a first image; - transmits a prompt including a specification of the execution LoRA of the element-tuned model and the acquired first image; this causes the generation AI to operate using the LoRA of the element-tuned model and acquire a second image; - transmits a prompt including a specification of the execution LoRA of the environmentally tuned model and the acquired second image; this causes the generation AI to operate using the LoRA of the environmentally tuned model and acquire a third image; - acquires the acquired third image as an image showing the air condition of the target environment and displays it to the user.

[0264] In this way, in the information processing device 120 according to the eighth embodiment, the generation AI that has undergone additional learning for each category (configuration, element, environment) is operated in the order of the categories (configuration, element, environment). As a result, the information processing device 120 according to the eighth embodiment can more appropriately generate an image that intuitively expresses the state of air.

[0265] In the above description, configuration, element, and environment are listed as categories, but the number and types of categories are not limited to these. Furthermore, in the above description, one execution LoRA is specified for each category, but the number of execution LoRAs specified for each category is not limited to one and may be multiple. For example, as described in the sixth embodiment, multiple execution LoRAs may be specified for a specific category as a result of re-specifying the execution LoRA because it is determined that the sentence included in the prompt does not satisfy the environmental conditions assigned to the training data.

[0266] Ninth Embodiment A specific example of an image that intuitively expresses the state of air, generated by an image generation system including the information processing device 120 according to the sixth and eighth embodiments, will be described. Fig. 45 is a diagram showing an example of image generation processing by the image generation system.

[0267] The example of Figure 45 shows that the information processing device 120 has acquired the following environmental data: Date and time: 16:00 on July 25th Location: Karuizawa Temperature: 25°C Humidity: 60% Wind speed: 2 m / s, etc.

[0268] The example in Figure 45 shows how the information processing device 120 generates environmental terms such as "hot," "relatively clear visibility," "slightly strong wind," "sunset," and "summer resort" based on environmental data.

[0269] The example in Figure 45 shows how the information processing device 120 generates, based on the environmental language and environmental data, the following sentence to be included in the prompt: "A summer evening in Karuizawa. Under the intense sunlight, you can hear the chirping of cicadas. When you enter the shade of the trees, a refreshing breeze blows through, providing a little relief from the heat."

[0270] The example of Figure 45 shows that the information processing device 120 has identified the following image editing parameters to be included in the prompt based on the environmental data: Brightness: +15, Contrast: +10, Saturation: +15, Sharpness: +10, Blur: 2 to 3.

[0271] The example in Figure 45 shows that the information processing device 120: - specifies a sunset LoRA as the execution LoRA of the configuration-tuned model; - specifies a wind speed LoRA of the element-tuned model instead of the execution LoRA of the environment-tuned model; and - specifies a temperature LoRA as the execution LoRA of the element-tuned model.

[0272] In the example of Figure 45, for reference, symbols 4501 to 4503 indicate images obtained when the information processing device 120 operates the generation AI by separately including the execution LoRA for each specified category in the prompt.

[0273] 45, reference numeral 4510 denotes an image acquired when the information processing device 120 operates the generation AI by sequentially including the specification of the execution LoRA of each category in the prompt. As is clear from a comparison between the images indicated by reference numerals 4501 to 4503 and the image indicated by reference numeral 4510, the images generated are significantly different between: - when an image is generated by operating the execution LoRA of each category individually; and - when an image is generated by operating the execution LoRA of each category sequentially.

[0274] In this way, by categorizing multiple tuned models and running them in the appropriate order, it is possible to generate highly accurate images that intuitively represent the air condition.

[0275] Other Embodiments In the above embodiments, the information processing program is executed by one information processing device 120, but the information processing program may be executed by a plurality of information processing devices in cooperation with each other.

[0276] Note that the functions realized by the information processing device 120 by executing the information processing program are not limited to the functions described in the above embodiments. For example, some or all of the functions of the server device 110 and some or all of the functions of the server device 140 may be realized in the information processing device 120.

[0277] Although the embodiments have been described above, it will be understood that various changes in form and details can be made without departing from the spirit and scope of the claims.

[0278] This application claims priority based on Japanese Patent Application No. 2024-154254, filed on September 6, 2024, the entire contents of which are incorporated herein by reference.

[0279] 100: Image generation system 110: Server device 111: Generation AI 120: Information processing device 130_1 to 130_n: Sensor 1 to sensor n 200: Control unit 310: Communication control unit 320: Environmental data acquisition unit 330: Environmental language generation unit 340: Prompt generation unit 350: Output unit 2110: Action control language generation unit 2500: Image generation system 2511: Fine tuning unit 2540: Server device 2610: Image data collection unit 2620: Caption generation unit 2630: Learning data generation unit 2640: Additional learning unit 3100: Image generation system 3110: Tuned model 3210: Weight calculation unit 3610: Selection unit 3810: Weight calculation unit 4100 : Image generation system 4111 : Configuration fine tuning unit 4112 : Element fine tuning unit 4113 : Environment fine tuning unit 4210 : Configuration learning data 4220 : Element learning data 4230 : Environment learning data 4300 : Image generation system 4311 : Configuration tuned model 4312 : Element tuned model 4313 : Environment tuned model

Claims

1. An information processing device having a control unit, wherein the control unit: acquires environmental data including item values ​​of items that indicate the air condition of a target environment or item values ​​of items that affect the air condition of the target environment; generates environmental language by converting the environmental data into natural language including sensory expressions; generates prompts including words, sentences, or images that express images that co-occur from the environmental language; operates a generation AI using the prompt; and displays an image that indicates the air condition of the target environment, generated by the generation AI.

2. The information processing device according to claim 1, wherein the control unit generates a prompt including a phrase, a sentence, or an image that expresses an image that co-occurs from the environmental data.

3. The information processing device according to claim 1, wherein the control unit acquires image editing parameters corresponding to the environmental data or image editing parameters corresponding to a sentence included in the prompt, and generates the prompt including the image editing parameters.

4. The information processing device according to claim 1, wherein the control unit acquires image editing parameters corresponding to the identified style, and generates the prompt including the image editing parameters.

5. The information processing device according to claim 1, wherein the control unit, when displaying an image showing the air condition of the target environment, displays the image in comparison with an image showing the air condition of a comparison environment different from the target environment.

6. The information processing device described in claim 1, wherein the control unit generates a motion control language representing the movement of components based on the environmental data, generates the prompt including the motion control language, operates a generation AI using the prompt, and displays a moving image showing the air condition of the target environment generated by the generation AI, in which the components move according to the motion control language.

7. An information processing method in which a control unit of an information processing device executes the following process: acquires environmental data including item values ​​of items that indicate the air condition of a target environment or item values ​​of items that affect the air condition of the target environment; generates environmental language by converting the environmental data into natural language including sensory expressions; generates prompts including words, sentences, or images that express images that co-occur from the environmental language; operates a generation AI using the prompt; and displays an image generated by the generation AI that indicates the air condition of the target environment.

8. An information processing program for causing a control unit of an information processing device to execute the following process: acquire environmental data including item values ​​of items indicating the air condition of a target environment or item values ​​of items that affect the air condition of the target environment; generate environmental language by converting the environmental data into natural language including sensory expressions; generate prompts including words, sentences, or images that express images that co-occur from the environmental language; operate a generation AI using the prompts; and display an image generated by the generation AI that indicates the air condition of the target environment.

Images