Processing apparatus, processing method, and program

Abstract

To construct a multi-modal AI more easily.SOLUTION: A processing apparatus determines, based on an input by a user, an algorithm to be executed by a multi-modal AI, data to be processed by the executed algorithm, and an execution order of the algorithm.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a processing device, a processing method, and a program. [Background technology]

[0002] There is a technology called multimodal AI (Artificial Intelligence). Multimodal AI is an AI technology that processes multiple types of data, such as text and images. Multimodal AI can make more advanced decisions than single-modal AI, which processes only one type of data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6697159 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the cost and technical difficulty of building multimodal AI are very high, making it difficult to build. An object of the present invention is to provide a processing device, a processing method, and a program that can more easily construct a multimodal AI. [Means for solving the problem]

[0005] One aspect of the present invention is a processing device that determines algorithms and the order in which the algorithms are executed based on input from a user, and constructs a multimodal AI. [Effects of the Invention]

[0006] According to the present invention, multimodal AI can be constructed more easily. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram showing a configuration of a communication system 1 according to the present embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of a processing device 12 according to the present embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of a constructed multimodal AI. [Figure 4] 1 is a flowchart illustrating the construction of a multimodal AI according to the present embodiment. [Figure 5] 10 is a flowchart illustrating the correction of a multimodal AI according to the present embodiment. [Figure 6] This is an example of no-code display. [Figure 7] This is an example of no-code display. [Figure 8] 1 is a flowchart illustrating the execution of a multimodal AI according to the present embodiment. [Figure 9] FIG. 10 is a diagram showing a crime prevention multimodal AI4-1. [Figure 10] FIG. 10 is a diagram showing a store management multimodal AI4-2. [Figure 11] FIG. 10 is a diagram showing a farmland management multimodal AI4-3. [Figure 12] FIG. 10 is a diagram showing a health management multimodal AI4-4. DETAILED DESCRIPTION OF THE INVENTION

[0008] (Overall composition) Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. 1 is a diagram showing the configuration of a communication system 1 according to this embodiment. The communication system 1 performs communication between devices.

[0009] The communication system 1 includes a processing device 12, a processing device 13-1, a processing device 13-2, a user device 20, and a network 30. In the following description, when there is no need to distinguish between the processing device 13-1 and the processing device 13-2, they may be simply referred to as "processing device 13." Furthermore, the number of processing devices 13 included in the communication system 1 is not limited to two, and may be one, or three or more. The processing device 12, the processing device 13, and the user device 20 communicate with each other via the network 30. The processing device 12 constructs a multimodal AI. The processing contents of the processing device 12 and the processing device 13 will be described later. A user U of the user device 20 causes the processing device 12 to perform a predetermined process by transmitting an instruction from the user device 20 to the processing device 12.

[0010] The processing device 13 is a device that performs a predetermined process based on a predetermined algorithm. The processing device 12 controls the processing device 13 to cause the processing device 13 to perform the predetermined process.

[0011] The processing device 12 is, for example, a virtual machine realized in a cloud. The processing device 13 is, for example, a virtual machine realized in a cloud. The processing device 12 and the processing device 13 may be realized in the same cloud.

[0012] The user device 20 includes an input device 21, a processing device 22, and a display device .

[0013] The input device 21 accepts input of various data to the processing device 22 via a wired or wireless connection. The input device 21 inputs the input information to the processing device 22. The input device 21 is, for example, a mouse, a keyboard, or a touch panel.

[0014] The processing device 22 processes data input from the input device 21. The processing device 22 may also process data input from another system. The processing device 22 transmits and receives data to and from the processing device 12 via the network 30.

[0015] The display device 23 displays the data output by the processing device 22. The display device 23 is, for example, a display such as a liquid crystal display.

[0016] The input device 21 may include a sensor that recognizes the movements of the user U and a microphone that recognizes the voice of the user U. The display device 23 may also include smart glasses. The display device 23 may incorporate all or part of the input device 21.

[0017] The user U inputs instructions to the processing device 22 using the input device 21, and the processing device 22 transmits the instructions to the processing device 12 via the network 30, thereby controlling the processing device 12 and causing the processing device 12 to construct a multimodal AI. The user U may change the input content according to the content displayed on the display device 23 and cause the processing device 12 to construct a multimodal AI. The user U causes the processing device 12 to modify the multimodal AI using the input device 21. The user U may also cause the processing device 12 to modify the multimodal AI in accordance with the content displayed on the display device 23. The processing device 12 executes multimodal AI based on data input from the user U or other systems, and generates analysis results by the multimodal AI.

[0018] The following describes in detail the processing of the processing device 12. Fig. 2 is a diagram showing the configuration of the processing device 12 according to this embodiment. The processing device 12 includes an acquisition unit 121, a multimodal AI construction unit 122, a multimodal AI correction unit 123, a multimodal AI execution unit 124, a recording unit 125, an output unit 126, and a storage unit 129.

[0019] The storage unit 129 stores algorithm plugins. The storage unit 129 stores the algorithm plugins in a predetermined format. The plugins correspond to the algorithms executed by the respective processing devices 13, and correspond to, for example, the interfaces of the algorithms executed by the respective processing devices 13. By using plugins, differences in the specifications of each algorithm are absorbed, and any algorithm can be handled in an abstract manner as "calling," "inputting data," and "obtaining an output result."

[0020] The algorithm plug-in is created using a framework or library for constructing an interface for the algorithm, and is, for example, code for HTTP access to the REST-API provided by the processing device 13. An algorithm plugin may correspond to one specific algorithm or to multiple algorithms.

[0021] Note that algorithms are stored in the processing device 13, and the processing device 12 controls the algorithms stored in the processing device 13 by accessing the processing device 13. The algorithms stored in the processing device 13 include, for example, single-modal AI. The algorithms stored in the processing device 13 include, for example, multi-modal AI.

[0022] The processing device 12 causes the processing device 13 to process data, for example, by the following procedure: The processing device 12 inputs data to the processing device 13 via a corresponding plug-in. The processing device 13 then executes a stored algorithm to process the input data. The processing device 12 obtains the processing result of the algorithm from the processing device 13 via the corresponding plug-in. This allows the processing device 12 to cause the processing device 13 to process data regardless of the interface of the algorithm.

[0023] The data input to the processing device 13 may be data indicating a storage location of data to be processed. The data indicating the storage location of data is, for example, data indicating a bucket path in a cloud. The format of the data indicating the storage location of data is, for example, JSON. Similarly, the processing result of an algorithm may be data indicating a storage location of the processing result.

[0024] (Multimodal AI construction) The acquisition unit 121 acquires a multimodal AI construction instruction from the user device 20. The user U uses the user device 20 to send the multimodal AI construction instruction to the processing device 12. The multimodal AI construction unit 122 constructs a multimodal AI by combining algorithms corresponding to algorithm plugins stored in the storage unit 129 based on the multimodal AI construction instruction.

[0025] The multimodal AI construction instructions input to the processing device 12 include, for example, information on algorithms to be executed by the multimodal AI to be constructed, among the algorithms executed by the processing device 13. The multimodal AI construction instructions input to the processing device 12 include, for example, information on the data to be processed and the order of the algorithms to be executed by the multimodal AI to be constructed. The multimodal AI construction instructions input to the processing device 12 include, for example, information on the weighting of the execution results of the algorithms that make up the multimodal AI to be constructed. The multimodal AI construction instructions input to the processing device 12 include, for example, information on the parameter values ​​of each algorithm to be executed by the multimodal AI.

[0026] Based on the multimodal AI construction instructions acquired by the acquisition unit 121, the multimodal AI construction unit 122 determines the algorithms to be executed, the data to be processed by the algorithms, the order in which the algorithms will be executed, and the weighting of the results of the algorithm execution, and determines the parameter values ​​of each algorithm to be executed, thereby constructing a multimodal AI.

[0027] The multimodal AI constructor 122 records the constructed multimodal AI in the storage unit 129.

[0028] FIG. 3 is a diagram illustrating an example of processing performed by a multimodal AI to be constructed. Here, the multimodal AI construction instruction acquired by the acquisition unit 121 indicates that the algorithms to be executed by the multimodal AI to be constructed are algorithm A, algorithm B, algorithm C, algorithm D, and algorithm E. The multimodal AI construction instruction acquired by the acquisition unit 121 indicates instructions to have algorithm A process first input data, algorithm B process second input data, algorithm C process third input data, algorithm D process the processing results of algorithms A and B and output the processing results as first output data, and algorithm E process the processing results of algorithms B and C and output the processing results as second output data. The multimodal AI construction unit 122 combines plug-ins based on the multimodal AI construction instruction to construct a multimodal AI that performs the processing shown in FIG. 3.

[0029] 4 is a flowchart showing the construction of a multimodal AI according to this embodiment. The acquisition unit 121 acquires a multimodal AI construction instruction from the user U (step S11). The multimodal AI construction unit 122 constructs a multimodal AI from plugins stored in the storage unit 129 based on the multimodal AI construction instruction (step S12). The recording unit 125 records the constructed multimodal AI in the storage unit 129 (step S13).

[0030] As described above, the processing device 12 can construct a multimodal AI. The multimodal AI constructed here is constructed by combining pre-created plug-ins, and therefore can be constructed more easily than conventional multimodal AI. Furthermore, the processing device 12 can construct a multimodal AI that causes appropriate algorithms to perform processing in an appropriate order.

[0031] (Multimodal AI correction) The acquisition unit 121 acquires a multimodal AI correction instruction from the user device 20. The user U inputs the multimodal AI correction instruction to the processing device 12 using the user device 20.

[0032] The multimodal AI modification instructions input to the processing device 12 include information identifying the multimodal AI to be modified. The multimodal AI modification instructions input to the processing device 12 include, for example, information on algorithms to be changed, added, or deleted in the multimodal AI to be modified. The multimodal AI modification instructions input to the processing device 12 include, for example, information on the execution order of the algorithms of the multimodal AI to be modified. The multimodal AI modification instructions input to the processing device 12 include, for example, information on weighting of the execution results of the algorithms executed by the multimodal AI to be modified. The multimodal AI modification instructions input to the processing device 12 include, for example, information on the parameter values ​​of each algorithm executed by the multimodal AI.

[0033] Based on the acquired multimodal AI modification instruction, the multimodal AI modification unit 123 modifies the multimodal AI stored in the storage unit 129. Based on the multimodal AI modification instruction, the multimodal AI modification unit 123 changes, for example, the algorithm executed by the multimodal AI to a different algorithm.

[0034] 5 is a flowchart showing the process of modifying a multimodal AI according to this embodiment. The acquisition unit 121 acquires a multimodal AI modification instruction from the user U (step S21). The multimodal AI modification unit 123 modifies the multimodal AI stored in the storage unit 129 based on the multimodal AI modification instruction (step S22).

[0035] As described above, the processing device 12 can modify the multimodal AI. In conventional multimodal AIs, it is difficult to understand the internal processing, and even when a new algorithm capable of performing highly accurate analysis emerges, it is difficult to replace the processing in the multimodal AI with the processing of the new algorithm. In the communication system 1 according to this embodiment, the algorithm executed by the multimodal AI can be easily changed as described above, making it easier to perform processing incorporating the new algorithm.

[0036] (No-code / Low-code) The processing device 12 may be configured so that the user U can construct and modify the multimodal AI using no-code or low-code. In this case, the processing device 12 transmits information about algorithms corresponding to algorithm plugins stored in the storage unit 129 to the processing device 22. In this case, in constructing the multimodal AI, the processing device 22 of the user device 20 causes the display device 23 to display the algorithms corresponding to the algorithm plugins stored in the storage unit 129. The processing device 22 also causes the display device 23 to display input data settings. The user U uses the input device 21 to set input data for the multimodal AI displayed on the display device 23. The processing device 22 also causes the display device 23 to display settings between the input data and algorithms. The user U uses the input device 21 to set the relationship between the input data and algorithms displayed on the display device 23. The user U uses the input device 21 to select a displayed algorithm, thereby selecting an algorithm to be executed by the multimodal AI to be constructed. The user U uses the input device 21 to rearrange the order of the algorithms displayed on the display device 23, thereby setting the data to be processed by the algorithm and the execution order. The user U sets the weighting of the execution results of the algorithms by inputting weighting information for each displayed algorithm using the input device 21. The user U sets the parameters of each algorithm by inputting the parameters of each algorithm displayed on the display device 23 using the input device 21.

[0037] The settings made by the user U using the input device 21 are transmitted to the processing device 12. The processing device 12 constructs a multimodal AI based on the received settings. This allows the user U to construct a multimodal AI with no code or low code.

[0038] Furthermore, when modifying a multimodal AI, the processing device 22 causes the display device 23 to display algorithms corresponding to the multimodal AIs and algorithm plugins stored in the storage unit 129. The user U uses the input device 21 to select a multimodal AI displayed on the display device 23. The processing device 22 then causes the display device 23 to display the algorithms to be executed by the selected multimodal AI, the data to be processed, and the execution order. The processing device 22 also causes the display device 23 to display algorithms corresponding to the algorithm plugins stored in the storage unit 129. The user U uses the input device 21 to select an algorithm to be changed, thereby changing the algorithm to be executed by the multimodal AI. The user U uses the input device 21 to select an algorithm to be added or deleted, thereby adding or deleting an algorithm to be executed by the multimodal AI. The user U uses the input device 21 to select a weighting for the algorithm and input the changed weighting, thereby changing the weighting of the execution result of the algorithm. The user U uses the input device 21 to select a parameter of the algorithm to be changed and input the changed parameter, thereby changing the parameter of the algorithm.

[0039] Setting changes made by the user U using the input device 21 are transmitted to the processing device 12. The processing device 12 modifies the multimodal AI based on the received settings. This allows the user U to modify the multimodal AI with no code or low code.

[0040] Figure 6 is an example of a no-code display. The multimodal AI shown in Figure 6 executes facial expression analysis algorithm A, voice analysis algorithm A, chat analysis algorithm A, and analysis algorithm A. By using the input device 21 to select input settings, the user U specifies a storage path in the memory unit 129 where the files to be input to the algorithm are located, and the input to the multimodal AI is determined. The user U uses the input device 21 to select an input data inter-algorithm setting and set the input data to be input to the algorithm. For example, the user U sets which algorithm the data input to the multimodal AI will be input to by setting a classification rule between part of the input data file name or extension and the destination algorithm.

[0041] The user U uses the input device 21 to select a facial expression analysis algorithm A, a voice analysis algorithm A, a chat analysis algorithm A, or an analysis algorithm as the algorithm to be executed by the multimodal AI, thereby determining the algorithm to be executed by the multimodal AI.

[0042] The user U uses the input device 21 to input the connection relationships of each algorithm, thereby determining the order in which the algorithms are executed by the multimodal AI.

[0043] The user U uses the input device 21 to input or select parameters for the facial expression analysis algorithm A, the voice analysis algorithm A, the chat analysis algorithm A, and the analysis algorithm, thereby setting the parameter values ​​for each algorithm to be executed by the multimodal AI.

[0044] The user U uses the input device 21 to input or select the weighting settings for the analysis algorithms, thereby setting the weightings for the analysis algorithms of the execution results of the facial expression analysis algorithm A, the voice analysis algorithm A, and the chat analysis algorithm A that make up the multimodal AI.

[0045] The flow information of the multimodal AI created here is sent to the processing device 12 and processed by a parser that corresponds to the data format of the flow information.

[0046] Figure 7 shows an example of modifying a multimodal AI using no-code. Plugins corresponding to speech analysis algorithm B, speech analysis algorithm C, and speech analysis algorithm D are stored in memory unit 129, and speech analysis algorithm B, speech analysis algorithm C, and speech analysis algorithm D are displayed.

[0047] User U can use the input device 21 to select voice analysis algorithm A in the multimodal AI, and then select voice analysis algorithm B, thereby changing the algorithm for analyzing voice executed by the multimodal AI from voice analysis algorithm A to voice analysis algorithm B. At this time, user U inputs or selects parameters for voice analysis algorithm B, thereby setting the parameter values ​​of voice analysis algorithm B executed by the multimodal AI. The system may be configured so that settings before the change can be retained when the algorithm is changed. Changes made using no-code are reflected in the multimodal AI stored in the memory unit 129 by the processing device 12. The multimodal AI shown in FIG. 7 executes facial expression analysis algorithm A, voice analysis algorithm B, chat analysis algorithm A, and analysis algorithm.

[0048] For example, when manually configuring an algorithm provided by an external system, the tools for creating the configuration file and the notation for the configuration file are difficult to understand, which requires time and money to master. By creating a multimodal AI flow using no-code or low-code and using a parser to process the settings for the created flow, it is possible to wrap complex setting processes. This allows the communication system 1 to more easily and intuitively build and modify multimodal AI.

[0049] (Multimodal AI execution) The acquisition unit 121 acquires data to be input to the multimodal AI. The acquisition unit 121 acquires data input from the user device 20 or an external system. The external system is, for example, a sensor, and the acquisition unit 121 acquires data measured by the sensor.

[0050] The multimodal AI execution unit 124 inputs the data acquired by the acquisition unit 121 into the multimodal AI stored in the storage unit 129, causing the data to be processed. The data is input into the multimodal AI, which then causes the processing device 13 to execute an algorithm, and the final analysis result is output.

[0051] The output unit 126 outputs the final analysis result obtained by the multimodal AI to the outside. The output unit 126 outputs the final analysis result to, for example, the processing device 22. The processing device 22 outputs the received analysis result to the display device 23 for display. The output unit 126 outputs the analysis result to, for example, another system. Here, the system from which the acquisition unit 121 acquires data may be the same as the system to which the output unit 126 outputs data.

[0052] In addition, when the output unit 126 allows the user U to construct and modify a multimodal AI using the above-mentioned no-code or low-code, it outputs to the processing device 22 information on the algorithm and multimodal AI corresponding to the algorithm plugin stored in the memory unit 129.

[0053] The output unit 126 may output not only the final analysis result by the multimodal AI, but also the processing results of each algorithm executed by the multimodal AI.

[0054] FIG. 8 is a flowchart showing the execution of the multimodal AI according to this embodiment. The acquisition unit 121 acquires data from a user U or an external system (step S31). The multimodal AI execution unit 124 executes the multimodal AI and inputs data into the multimodal AI to output an analysis result (step S32). At this time, the multimodal AI may output not only the final analysis result but also the processing results of each algorithm as intermediate processing results. The output unit 126 outputs the final analysis result (step S33).

[0055] The following describes a procedure for causing a multimodal AI to execute the process shown in Figure 3 as an example of multimodal AI execution. The multimodal AI execution unit 124 inputs first input data to the processing device 13-A executing algorithm A via a plugin corresponding to algorithm A, causes the processing device 13-A to process the first input data, and obtains the processing result by the processing device 13-A via the plugin corresponding to algorithm A. The multimodal AI execution unit 124 also inputs second input data to the processing device 13-B executing algorithm B via a plugin corresponding to algorithm B, causes the processing device 13-B to process the second input data, and obtains the processing result by the processing device 13-B via the plugin corresponding to algorithm B. The multimodal AI execution unit 124 also inputs third input data to the processing device 13-C executing algorithm C via a plugin corresponding to algorithm C, causes the processing device 13-C to process the third input data, and obtains the processing result by the processing device 13-C via the plugin corresponding to algorithm C.

[0056] The multimodal AI execution unit 124 inputs the processing results by the processing device 13-A and the processing results by the processing device 13-B to the processing device 13-D that executes algorithm D via a plug-in corresponding to algorithm D, causes the processing device 13-D to process the processing results by the processing device 13-A and the processing results by the processing device 13-B, and acquires the processing results by the processing device 13-D (first output data) via the plug-in corresponding to algorithm D. In addition, the multimodal AI execution unit 124 inputs the processing results by the processing device 13-B and the processing results by the processing device 13-C to the processing device 13-E that executes algorithm E via a plug-in corresponding to algorithm E, causes the processing device 13-E to process the processing results by the processing device 13-B and the processing results by the processing device 13-C, and acquires the processing results by the processing device 13-E (second output data) via the plug-in corresponding to algorithm E.

[0057] As described above, communication system 1 can perform analysis using multimodal AI. When output unit 126 outputs the processing results of each algorithm executed by the multimodal AI, it is possible to monitor intermediate processing data in the multimodal AI, and the intermediate processing data can be used as a criterion for selecting an algorithm to change.

[0058] Furthermore, the storage unit 129 stores multiple multimodal AIs, and the processing device 22 can select a multimodal AI and modify the selected multimodal AI. The processing device 22 can also select a multimodal AI and cause the processing device 12 to execute the selected multimodal AI. Therefore, the processing device 12 can handle multiple multimodal AIs in parallel at the same time.

[0059] In the examples shown in Figures 3, 6, and 7, the multimodal AI has the algorithm perform two-stage processing, but it may also perform three or more stages of processing. That is, in the examples shown in Figures 3, 6, and 7, data input to the multimodal AI is processed by an algorithm, and the processing result of one algorithm is processed by another algorithm to output a final analysis result. However, data input to the multimodal AI may be processed by a first algorithm, the processing result of the first algorithm may be processed by a second algorithm different from the first algorithm, the processing result of the second algorithm may be processed by a third algorithm different from the first and second algorithms, and the processing result may be further processed by a fourth algorithm, a fifth algorithm, and so on to output a final analysis result.

[0060] (Example) A specific example of a multimodal AI will be described below. Figure 9 is a diagram showing a crime prevention multimodal AI 4-1. The crime prevention multimodal AI 4-1 executes a combination of an image analysis algorithm 41, a sound analysis algorithm 42, and an integration algorithm 51-1. An image input to the crime prevention multimodal AI 4-1 is analyzed by the image analysis algorithm 41, and an image analysis result is generated. An audio input to the crime prevention multimodal AI 4-1 is analyzed by the sound analysis algorithm 42, and a sound analysis result is generated. The integration algorithm 51-1 integrates the image analysis result and the sound analysis result to generate a final analysis result. For example, by inputting images captured by a security camera and recorded audio into the security multimodal AI 4-1, analysis can be performed using two types of data: images and audio.

[0061] FIG. 10 illustrates the store management multimodal AI 4-2. The store management multimodal AI 4-2 executes a combination of an image analysis algorithm 41, a voice analysis algorithm 42, a weight analysis algorithm 43, a position analysis algorithm 44, and an integration algorithm 51-2. Images input to the store management multimodal AI 4-2 are analyzed by the image analysis algorithm 41 to generate an image analysis result. Voice input to the store management multimodal AI 4-2 is analyzed by the voice analysis algorithm 42 to generate a voice analysis result. Weight measurements input to the store management multimodal AI 4-2 by a weight sensor are analyzed by the weight analysis algorithm 43 to generate a weight analysis result. Location information input to the store management multimodal AI 4-2 is analyzed by the location analysis algorithm 44 to generate a location analysis result. The integration algorithm 51-2 integrates the image analysis result, voice analysis result, weight analysis result, and location analysis result to generate a final analysis result.

[0062] For example, by inputting images taken in the store, audio recorded in the store, weights measured by weight sensors on store shelves, and the positions of product tags on products being sold into the store management multimodal AI 4-2, analysis can be performed using four types of data.

[0063] FIG. 11 is a diagram illustrating the farmland management multimodal AI 4-3. The farmland management multimodal AI 4-3 executes a combination of an image analysis algorithm 41, a temperature analysis algorithm 45, a humidity analysis algorithm 46, a luminosity analysis algorithm 47, a CO2 concentration analysis algorithm 48, and an integration algorithm 51-3. An image input to the farmland management multimodal AI 4-3 is analyzed by the image analysis algorithm 41, and an image analysis result is generated. A temperature input to the farmland management multimodal AI 4-3 is analyzed by the temperature analysis algorithm 45, and a temperature analysis result is generated. A humidity input to the farmland management multimodal AI 4-3 is analyzed by the humidity analysis algorithm 46, and a humidity analysis result is generated. A luminosity input to the farmland management multimodal AI 4-3 is analyzed by the luminosity analysis algorithm 47, and a luminosity analysis result is generated. A CO2 concentration input to the farmland management multimodal AI 4-3 is analyzed by the CO2 concentration analysis algorithm 48, and a CO2 concentration analysis result is generated. The integration algorithm 51-3 integrates the image analysis result, the temperature analysis result, the humidity analysis result, the photometric analysis result, and the CO2 concentration analysis result to generate a final analysis result.

[0064] For example, by inputting images taken on farmland, temperature measured on farmland, humidity measured on farmland, light intensity measured on farmland, and CO2 concentration measured on farmland into the Farmland Management Multimodal AI4-3, analysis can be performed using five types of data.

[0065] 12 is a diagram showing the health management multimodal AI 4-4. The health management multimodal AI 4-4 executes a combination of an image analysis algorithm 41, a sound analysis algorithm 42, a vibration analysis algorithm 49, a temperature analysis algorithm 45, a humidity analysis algorithm 46, a luminosity analysis algorithm 47, a CO2 concentration analysis algorithm 48, and an integration algorithm 51-4.

[0066] An image input to the health management multimodal AI 4-4 is analyzed by an image analysis algorithm 41, generating an image analysis result. An audio input to the health management multimodal AI 4-4 is analyzed by an audio analysis algorithm 42, generating an audio analysis result. An vibration input to the health management multimodal AI 4-4 is analyzed by a vibration analysis algorithm 49, generating a vibration analysis result. An temperature input to the health management multimodal AI 4-4 is analyzed by a temperature analysis algorithm 45, generating a temperature analysis result. An humidity input to the health management multimodal AI 4-4 is analyzed by a humidity analysis algorithm 46, generating a humidity analysis result. An intensity input to the health management multimodal AI 4-4 is analyzed by a luminosity analysis algorithm 47, generating a luminosity analysis result. An CO2 concentration input to the health management multimodal AI 4-4 is analyzed by a CO2 concentration analysis algorithm 48, generating a CO2 concentration analysis result. The integration algorithm 51-4 integrates the image analysis result, the sound analysis result, the vibration analysis result, the temperature analysis result, the humidity analysis result, the luminosity analysis result, and the CO2 concentration analysis result to generate a final analysis result.

[0067] For example, by inputting images taken in a certain space, audio recorded in a certain space, vibrations measured in a certain space, temperature measured in a certain space, humidity measured in a certain space, luminosity measured in a certain space, and CO2 concentration measured in a certain space into the health management multimodal AI 4-4, it is possible to analyze the health status of a person in a certain space using seven types of data.

[0068] The store management multimodal AI 4-2 shown in Fig. 10, the farmland management multimodal AI 4-3 shown in Fig. 11, and the health management multimodal AI 4-4 shown in Fig. 12 execute integrating algorithms 51-2 to 51-4 to integrate all three or more types of analysis results, but some of all the analysis results may be integrated. For example, the multimodal AI 4 may include an integration algorithm that integrates image analysis results and audio analysis results, and the integrated results may be further integrated with other algorithms to generate a final analysis result.

[0069] Other Embodiments One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0070] In the above explanation, the construction, modification, and execution of the multimodal AI are all performed by the processing device 12, but these may also be performed in separate systems. For example, a multimodal AI construction system, a multimodal AI modification system, and a multimodal AI execution system may be provided separately. The multimodal AI construction system constructs a multimodal AI based on input from user U. The constructed multimodal AI is stored in a storage device of the multimodal AI modification system. The multimodal AI modification system modifies the multimodal AI based on input from user U. The multimodal AI execution system imports the multimodal AI from the multimodal AI modification system and executes the imported multimodal AI.

[0071] The processing of the processing device 12 in the above-described embodiment may be implemented by a computer using software. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an operating system and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client. The program may be for implementing only a portion of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array). [Explanation of symbols]

[0072] 1 Communication system, 12 Processing device, 13 Processing device, 20 User device, 21 Input device, 22 Processing device, 23 Display device, 121 Acquisition unit, 122 Multimodal AI construction unit, 123 Multimodal AI correction unit, 124 Multimodal AI execution unit, 125 Recording unit, 126 Output unit, 129 Memory unit

Claims

1. Based on input from a user, the algorithm to be executed by the multimodal AI, the data to be processed by the algorithm to be executed, and the execution order of the algorithm are determined. Processing equipment.

2. determining a weighting of the results of the execution of the algorithm based on the input from the user; The processing device of claim 1 .

3. outputting data indicating a storage location of data to be processed by the algorithm to the algorithm; The processing device according to claim 1 or 2.

4. Modifying, deleting, or adding algorithms to be executed by the multimodal AI based on input from the user; The processing device according to claim 1 or 2.

5. The construction and modification of the multimodal AI is performed using no-code or low-code. The processing device according to claim 4 .

6. Inputting data into the multimodal AI to process the data; The processing device according to claim 1 or 2.

7. determining parameter values ​​in an algorithm based on input from the user; The processing device of claim 1 .

8. Based on input from a user, the algorithm to be executed by the multimodal AI, the data to be processed by the algorithm to be executed, and the execution order of the algorithm are determined. Processing method.

9. A program for causing a computer to execute the processing method according to claim 8.

Images