Learning program, inference program, and learning method

Abstract

To achieve the selection of a machine learning model capable of expecting highly accurate output to user input.SOLUTION: Inputting training data in which media data and text data are associated with a first machine learning model selected as a correct answer label from a plurality of first machine learning models based on an execution result obtained by executing a task corresponding to the text data on the media data and a correct answer rate for a distribution of a plurality of pieces of media data to which the media data belongs; Training a parameter of a second machine learning model that outputs a label of at least one first machine learning model from among the plurality of first machine learning models, using the media data and the text data in the training data as explanatory variables and the first machine learning model selected as the correct label as an objective variable.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning program, an inference program, and a learning method. [Background technology]

[0002] One type of multimodal AI (Artificial Intelligence) is a machine learning model that processes visual and linguistic information in an integrated manner, known as the Vision Language Model (VLM).

[0003] For example, systems such as GPT-4V attempt to handle general-purpose tasks, but there are cases where it is difficult to achieve the accuracy desired by users for some tasks. In such cases, fine-tuning a machine learning model for each task increases costs and resources in terms of time and economy.

[0004] For this reason, there is a need for technology that can select a machine learning model that is compatible with user input, such as media data or tasks, while minimizing fine-tuning, ideally zero-shot, and achieving improved accuracy.

[0005] As a technology that attempts to select a machine learning model in this way, a large-scale language model has been released that accepts prompts written in natural language from the user and outputs an API (Application Programming Interface) that can handle the task corresponding to the prompt. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-33826 [Patent Document 2] Japanese Patent Application Publication No. 2019-87229 [Patent Document 3] Patent Publication No. 2021-179859 [Patent Document 4] US Patent Application Publication No. 2019 / 0391956 [Non-patent literature]

[0007] [Non-Patent Document 1] Shishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. Gonzalez, “Gorilla: Large Language Model Connected with Massive APIs”, Submitted on 24 May 2023 [Non-patent document 2] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang, “HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face”, Submitted on 30 Mar 2023 (v1), last revised 3 Dec 2023 (this version, v4) Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the above-mentioned large-scale language models, it remains difficult to select a machine learning model that can be expected to produce highly accurate output for user input.

[0009] For example, the large-scale language model simply selects an API that can handle the task corresponding to the prompt entered by the user, and the accuracy of the selection is unknown until the API is executed. Furthermore, the large-scale language model selects an API without considering the characteristics of the media data entered by the user, so even if an API that is compatible with the task is selected, it does not necessarily mean that the selected API is compatible with the media data.

[0010] In one aspect, the present invention aims to provide a learning program, an inference program, and a learning method that enable the selection of a machine learning model that can be expected to produce highly accurate output for user input. [Means for solving the problem]

[0011] A learning program according to one aspect inputs training data in which media data and text data are associated with a first machine learning model selected from a plurality of first machine learning models as a correct label based on the execution result of a task corresponding to the text data being executed on the media data and the accuracy rate for the distribution of a plurality of media data to which the media data belongs, and causes a computer to execute a process of training parameters of a second machine learning model that outputs labels of one or more first machine learning models from the plurality of first machine learning models, using the media data and the text data of the training data as explanatory variables and the first machine learning model selected as the correct label as a target variable. [Effects of the Invention]

[0012] According to one embodiment, it is possible to select a machine learning model that can be expected to produce highly accurate output in response to user input. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of each device in the machine learning system. [Figure 2]FIG. 2 is a schematic diagram (1) showing an example of training data generation. [Figure 3] FIG. 3 is a schematic diagram (2) showing an example of generating training data. [Figure 4] FIG. 4 is a schematic diagram showing one aspect of the clustering result. [Figure 5] FIG. 5 is a schematic diagram (3) showing an example of generating training data. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of training a model selection model. [Figure 7] FIG. 7 is a flowchart (1) showing the procedure of the generation process. [Figure 8] FIG. 8 is a flowchart (2) showing the procedure of the generation process. [Figure 9] FIG. 9 is a flowchart showing the procedure of the training process. [Figure 10] FIG. 10 is a schematic diagram showing an example of inference using the model selection model. [Figure 11] FIG. 11 is a schematic diagram showing an example of inference of a task execution model. [Figure 12] FIG. 12 is a flowchart showing the procedure of the inference process. [Figure 13] FIG. 13 is a schematic diagram (1) showing an example of an ensemble. [Figure 14] FIG. 14 is a schematic diagram (2) showing an example of an ensemble. [Figure 15] FIG. 15 is a diagram (1) showing an example of a hardware configuration. [Figure 16] FIG. 16 is a diagram (2) showing an example of the hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the learning program, inference program, and learning method according to the present disclosure will be described with reference to the accompanying drawings. Note that these embodiments merely illustrate one example or aspect, and the structures, actions, functions, properties, characteristics, methods, and uses according to the present disclosure are not limited by these examples. Furthermore, the embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0015] Example 1 <System configuration> Fig. 1 is a block diagram showing an example of the functional configuration of each device in a machine learning system 1. As shown in Fig. 1, the machine learning system 1 may include a learning device 10, an inference device 30, and a client terminal 50. Note that Fig. 1 shows an example in which one client terminal 50 is connected to one inference device 30, but any number of client terminals 50 may be connected.

[0016] The learning device 10 is a computer that provides a training function for training a second machine learning model that selects one or more from a plurality of first machine learning models that perform a task specified by text on media data.

[0017] Hereinafter, the first machine learning model may be referred to as the "task execution model," and the second machine learning model may be referred to as the "model selection model."

[0018] The inference device 30 is a computer that provides a model selection function that selects a task execution model using a trained model selection model. For example, the inference device 30 may be provided with not only the model selection function but also a model execution function that executes the task execution model selected by the model selection function, all packaged together.

[0019] For example, the inference device 30 can provide the above-described model selection function and the above-described model execution function as a cloud service by executing a SaaS (Software as a Service) type application. Also, the above-described model selection function and the above-described model execution function may be provided on-premise.

[0020] The client terminal 50 is a terminal device that receives the above-described model execution function. For example, the client terminal 50 may be realized by any computer, such as a personal computer, a smartphone, a tablet terminal, or a wearable terminal.

[0021] 1 illustrates an example in which the learning device 10 and the inference device 30 are implemented as separate devices, but they may also be integrated into one device. Furthermore, while the above-described model selection function and model execution function are provided in a client-server system, this is not limiting. For example, each function may be executed standalone by software running on the client terminal 50 causing the client terminal 50 to execute processing corresponding to the model selection function and model execution function.

[0022] <Configuration of learning device 10> Next, a functional configuration of the learning device 10 according to this embodiment will be described. For example, Fig. 1 shows a block diagram related to the training function of the learning device 10.

[0023] 1, learning device 10 includes a dataset group storage unit 11, a machine learning model group storage unit 12, a generation unit 13, a training dataset storage unit 14, and a training unit 15. Note that learning device 10 may also include functional units other than those shown in FIG. 1, such as a communication control unit that provides a communication interface.

[0024] The dataset group storage unit 11 is a functional unit that stores a set of datasets corresponding to various media data recognition tasks. The "media data" referred to here may include image data, audio data, and the like.

[0025] Below, we will cite an example in which a VLM that processes visual and linguistic information in an integrated manner is selected as just one example of a usage scenario in which the above-mentioned model selection function is applied. In this case, the media data recognition task performed by the VLM may be any image recognition task. Note that the above-mentioned usage scenario is merely an example, and it goes without saying that the above-mentioned model selection function can also be applied to other usage scenarios, such as scenarios in which a multimodal AI that processes auditory and linguistic information in an integrated manner is selected.

[0026] In one embodiment, the datasets stored in the dataset group storage unit 11 may include datasets such as CIFAR100, MSCOCO, and VisualGenome that are published on a network. CIFAR is an abbreviation for "Canadian Institute For Advanced Research." MSCOCO is an abbreviation for "Microsoft (registered trademark) Common Objects in Context."

[0027] The machine learning model group storage unit 12 is a functional unit that stores a collection of task execution models corresponding to various media data recognition tasks. In one embodiment, the task execution models stored in the machine learning model group storage unit 12 may include machine learning models that execute all image recognition tasks that can be expected to be used as a VLM. For example, the machine learning model group storage unit 12 may store machine learning models that execute various image recognition tasks such as VQA (Visual Question Answering), object recognition, and segmentation.

[0028] The generation unit 13 is a processing unit that uses a set of datasets stored in the dataset group storage unit 11 to generate a dataset to be used for training the model selection model.

[0029] Hereinafter, in order to distinguish between the labels of the datasets stored in the dataset group storage unit 11 and the datasets used to train the model selection model, the former may be referred to as the "raw dataset" and the latter may be referred to as the "training dataset." Furthermore, each of the samples included in the training dataset may be referred to as "training data."

[0030] In one embodiment, the generation unit 13 executes a machine learning model that is expected to be used for an actual similar task and obtains the obtained results. Here, the material dataset stored in the dataset group storage unit 11 may contain tens of thousands of images, and multiple image recognition tasks may be executed for each image.

[0031] As just one example, the generation unit 13 inputs the mth image data into the nth task execution model in units of the number K of task types that can be expected to be used as a VLM, the number I of material datasets corresponding to the kth task, the number M of samples included in the ith material dataset, and further, the number N of task execution models corresponding to the kth task, thereby obtaining the execution results output by the nth task execution model.

[0032] Then, generation unit 13 sets a task execution model to be used as a correct label from among the N task execution models for each of the M pieces of image data included in each of the I material datasets. At this time, generation unit 13 sets the correct label based on the execution result of each of the N task execution models and the accuracy, for example, accuracy rate, of the N task execution models for the material dataset that includes the image data among the I material datasets. In this way, training data in which the mth image data, the kth task, and the correct label are associated with each other is obtained.

[0033] FIG. 2 is a schematic diagram (1) showing an example of generating training data. For example, FIG. 2 illustrates a scene in which a correct label is set to image data I11 used in an object recognition task. Furthermore, FIG. 2 shows correct or incorrect answers as examples of execution results E1 to EN output by N task execution models 130A to 130N to which image data I11 belonging to material dataset i has been input. Although not shown in the figure, it is assumed that text corresponding to the object recognition task is input to each of the N task execution models 130A to 130N.

[0034] In the example shown in FIG. 2, the generation unit 13 extracts three task execution models 130A, 130C, and 130N that have correct execution results from among the N task execution models 130A to 130N. Of these three task execution models 130A, 130C, and 130N, the generation unit 13 extracts the task execution model 130C that has the highest accuracy rate for the entire dataset i. As a result, among the task execution models that can correctly solve the object recognition task of the image data I11, the task execution model 130C that can more appropriately solve the object recognition task can be set as the correct label. As a result, training data 140 is obtained in which modal 1 "image data I11," modal 2 "object recognition task," and the correct label "task execution model 130C" are associated. Note that text corresponding to the object recognition task may be associated with the training data 140.

[0035] Here, Figure 2 shows an example of using the correct or incorrect output of each of the N task execution models as an example of the execution results of the N task execution models, but it is also possible to use the confidence of the label output by each of the N task execution models.

[0036] Fig. 3 is a schematic diagram (2) showing an example of generating training data. As in Fig. 2, Fig. 3 also shows a schematic diagram of a scene in which a correct label is set to image data I11 used in an object recognition task. Furthermore, Fig. 3 shows the certainty of the label as an example of execution results E1 to EN output by N task execution models 130A to 130N to which image data I11 belonging to material dataset i has been input.

[0037] In the example shown in FIG. 3, the generation unit 13 extracts two task execution models 130A and 130C from the N task execution models 130A to 130N, whose confidence levels are equal to or greater than a threshold, for example, 0.8. Of these two task execution models 130A and 130C, the generation unit 13 extracts the task execution model 130C that has the highest accuracy rate for the entire data set i. This also makes it possible to set the task execution model 130C, which is more likely to solve the object recognition task of the image data I11, as the correct label among the task execution models that can correctly solve the object recognition task of the image data I11. As a result, training data 140 is obtained in which modal 1 "image data I11," modal 2 "object recognition task," and the correct label "task execution model 130C" are associated with each other.

[0038] Furthermore, in Figures 2 and 3, the material dataset i to which image data I11 belongs is given as an example of a population for calculating the accuracy rate, which is an index of accuracy, but the population for calculating the accuracy rate is not limited to this.

[0039] For example, the generation unit 13 can cluster the I material data sets by image feature amounts. For such image feature amounts, embedding vectors obtained by inputting each of the M image data sets included in each of the I material data sets into an image encoder can be used.

[0040] Figure 4 is a schematic diagram showing one aspect of the clustering results. In Figure 4, a portion of the embedding space generated by the image encoder is excerpted, and image data belonging to material dataset i and image data belonging to material dataset j among I material datasets are excerpted and plotted. Furthermore, in Figure 4, the plot corresponding to image data I11 shown in Figures 2 and 3 is highlighted in bold.

[0041] For example, in the example shown in Fig. 4, material data set i and material data set j are clustered into two clusters C1 and C2. Clusters C1 and C2 obtained in this way have distributions of similar image features, so even if the material data sets contain a mixture of image data with different features, it is possible to reconstruct a population that functions effectively in calculating the accuracy rate.

[0042] Fig. 5 is a schematic diagram (3) showing an example of generating training data. For example, Fig. 5 also shows a schematic diagram of a scene in which a correct label is set to image data I11 used in an object recognition task, similar to Fig. 2. Furthermore, Fig. 5 shows correct or incorrect answers as examples of execution results E1 to EN output by the N task execution models 130A to 130N to which image data I11 belonging to cluster C1 shown in Fig. 4 is input.

[0043] In the example shown in FIG. 5, the generation unit 13 also extracts three task execution models 130A, 130C, and 130N that have correct execution results from among the N task execution models 130A to 130N. Here, of the three task execution models 130A, 130C, and 130N, the generation unit 13 extracts the task execution model 130N that has the highest accuracy rate for cluster C1. As a result, among the task execution models that can correctly solve the object recognition task of the image data I11, the task execution model 130N that can most appropriately solve the object recognition task can be set as the correct label. As a result, training data 140 is obtained in which modal 1 "image data I11," modal 2 "object recognition task," and the correct label "task execution model 130N" are associated.

[0044] In this way, a set of training data generated according to the generation method exemplified in FIGS. 2 to 5 is stored in the training data set storage unit 14 as a training data set.

[0045] Returning to the explanation of Fig. 1, the training unit 15 is a processing unit that trains a model selection model using the training dataset stored in the training dataset storage unit 14. Here, the model selection model may be realized by a VLM, just as an example.

[0046] In one embodiment, the training unit 15 trains the parameters of the model selection model by using the image data and text data included in the training data as explanatory variables of the model selection model and the correct answer label as a target variable of the model selection model.

[0047] Fig. 6 is a schematic diagram showing an example of training a model selection model. For example, Fig. 6 excerpts a scene in which training data 140 including modal 1 "image data I11", modal 2 "text data T11", and correct answer label "task execution model 130N" is used as one aspect of training using a training dataset.

[0048] As shown in Fig. 6, an image encoder 100T and a text encoder 100T are connected to the model selection model 100. For example, the model selection model 100 may be realized by Related Art 1 described below. Furthermore, the image encoder 100T may be realized by Related Art 2 described below. Furthermore, the text encoder 100T may be realized by Related Art 3 described below.

[0049] Related technology 1: Jean-Baptiste Alayrac et al. “Flamingo: a Visual Language Model for Few-Shot Learning”, Submitted on 29 Apr 2022 (v1), last revised 15 Nov 2022 (this version, v2) Related technology 2: openai / CLIP: CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image (github.com) Related Technology 3: Hugo Touvron et al. “LLaMA: Open and Efficient Foundation Language Models,” Submitted on 27 Feb 2023

[0050] With this configuration, image data I11 from the training data 140 is input to the image encoder 100M of the model selection model 100, and text data T11 is input to the text encoder 100T of the model selection model 100. The model selection model 100, to which the image data I11 and text data T11 have been input, outputs a task execution model 130X. In this case, the model selection model 100 updates its parameters to minimize the loss between the output of the model selection model 100, "task execution model 130X," and the correct label, "task execution model 130N."

[0051] Information about the trained model selection model 100 obtained in this manner may be stored in any storage, for example, the model selection model storage unit 32 of the inference device 30. For example, the model selection model storage unit 32 may include hyperparameters related to the layer structure of the model selection model 100, and parameters related to the objective function, such as the weights and biases of each layer.

[0052] <Processing flow> Next, a description will be given of the flow of processing by the learning device 10 according to this embodiment. Here, (1) generation processing executed by the learning device 10 will be described, followed by (2) training processing.

[0053] (1) Generation process 7 and 8 are flowcharts (1) and (2) showing the steps of the generation process. This process can be executed at any timing after each data is prepared in the dataset group storage unit 11 and the machine learning model group storage unit 12.

[0054] 7, the generation unit 13 executes loop processing 1, which repeats the processes from step S101 to step S106 described below a number of times corresponding to the number K of task types that can be expected to be used as a VLM. Note that the processes from step S101 to step S106 described below may be executed in parallel for each of the K task types.

[0055] Furthermore, generation unit 13 executes loop process 2, which repeats the process of step S101 below a number of times corresponding to the number I of material data sets corresponding to the k-th task. Note that the process of step S101 below may be executed in parallel for each of the I material data sets.

[0056] Furthermore, the generation unit 13 executes loop processing 3, which repeats the following step S101 a number of times corresponding to the number M of samples included in the i-th material data set. Note that the processing of the following step S101 may be executed in parallel for each of the M samples.

[0057] Furthermore, the generation unit 13 executes loop processing 4, which repeats the following step S101 a number of times corresponding to the number N of machine learning models corresponding to the k-th task. Note that the processing of the following step S101 may be executed in parallel for each of the N machine learning models.

[0058] That is, the generation unit 13 inputs the mth media data to the nth machine learning model, and thereby acquires the execution result output by the nth machine learning model (step S101).

[0059] By repeating this loop process 4, execution results are obtained for each of the N task execution models. Furthermore, by repeating loop process 3, execution results of the N task execution models are obtained for each of the M samples. Furthermore, by repeating loop process 2, execution results of the N task execution models are obtained for each of the M samples included in the I material datasets.

[0060] Next, the generation unit 13 clusters the I material data sets by image feature amounts, as shown in FIG. 8 (step S102).

[0061] Thereafter, the generation unit 13 executes a loop process 5 in which the processes from the following step S103 to the following step S106 are repeated a number of times corresponding to the number C of clusters obtained as a result of the clustering in step S102. Note that the processes from the following step S103 to the following step S106 may be executed in parallel for each of the C clusters.

[0062] Furthermore, the generation unit 13 executes loop processing 6, which repeats the processing of step S103 below a number of times corresponding to the number N of machine learning models corresponding to the k-th task. Note that the processing of step S106 below may be executed in parallel for each of the N machine learning models.

[0063] That is, the generation unit 13 calculates the accuracy rate of the nth machine learning model for the cth cluster (step S103).

[0064] By repeating this loop process 6, the accuracy rate for the c-th cluster is obtained for each of the N machine learning models.

[0065] Thereafter, the generation unit 13 executes a loop process 7 in which the processes from step S104 to step S106 below are repeated a number of times corresponding to the number P of samples included in the c-th cluster. Note that the processes from step S104 to step S106 below may be executed in parallel for each P samples.

[0066] That is, the generation unit 13 extracts a machine learning model from among the N machine learning models for which the execution result of the pth sample is correct (step S104). Furthermore, the generation unit 13 extracts a machine learning model from among the machine learning models extracted in step S104 that has the highest accuracy rate for the cth cluster (step S105).

[0067] Then, the generation unit 13 sets the machine learning model extracted in step S105 to the correct answer label of the training data of the p-th sample and the k-th task (step S106).

[0068] By repeating this loop process 7, a correct label is set for each of the P samples, resulting in P training data.

[0069] Furthermore, by repeating the loop process 1, P pieces of training data are generated for each of the K tasks, resulting in a complete training data set.

[0070] (2) Training process 9 is a flowchart showing the procedure of the training process. As shown in FIG. 9, the training unit 15 executes a loop process 1 in which the following steps S201 and S202 are executed until a predetermined termination condition, such as the number of epochs or convergence of the learning rate, is satisfied.

[0071] Furthermore, the training unit 15 executes loop processing 2, which executes the processing of step S201 and step S202 described below, a number of times corresponding to the number Q of training data included in the training dataset stored in the training dataset storage unit 14.

[0072] That is, the training unit 15 inputs the media data and text data included in the q-th training data to the model selection model (step S201). Then, the training unit 15 updates the parameters of the model selection model based on the loss between the output of the model selection model 100 in step S201 and the correct label set in the training data (step S202).

[0073] By repeating this loop process 2, training using the Q training data included in the training dataset is completed, resulting in the completion of one epoch. Furthermore, by repeating loop process 1, a trained model selection model is generated.

[0074] <Configuration of inference device 30> Next, the functional configuration of the inference device 30 according to this embodiment will be described. For example, Fig. 1 shows a block diagram of functions related to the model selection function and the model execution function of the inference device 30.

[0075] 1, the inference device 30 includes a reception unit 31, a model selection model storage unit 32, a model selection unit 33, a machine learning model group storage unit 34, and a model execution unit 35. Note that the learning device 10 may also include functional units other than those shown in FIG. 1, such as a communication control unit that provides a communication interface.

[0076] Of these, the model selection model storage unit 32 may store a model selection model 100 that has been trained by the training unit 15 of the learning device 10. Similar to the machine learning model storage unit 12, the machine learning model group storage unit 34 may store a set of task execution models corresponding to various media data recognition tasks.

[0077] The reception unit 31 is a processing unit that receives various types of information from the client terminal 50. As just one example, the reception unit 31 can receive a user request from the client terminal 50. When receiving such a user request, the reception unit 31 can receive user input such as designation of image data or text data via a GUI (Graphical User Interface).

[0078] The model selection unit 33 is a processing unit that uses the model selection model stored in the model selection model storage unit 32 to select one or more task execution models from the task execution models stored in the machine learning model group storage unit 34.

[0079] FIG. 10 is a schematic diagram showing an example of inference by a model selection model. For example, FIG. 10 shows an example in which a user input specifying image data 20M and text data 20T is received via a user request. As shown in FIG. 10, the model selection unit 33 inputs the image data 20M to the image encoder 100M of the model selection model 100, and inputs the text data 20T to the text encoder 100T of the model selection model 100. The model selection model 100 to which the image data 20M and text data 20T have been input outputs the label "task execution model 130." In this case, the model selection unit 33 selects the task execution model 130 from the task execution models stored in the machine learning model group storage unit 34.

[0080] The model execution unit 35 is a processing unit that uses the task execution model selected by the model selection unit 33 to execute processing on the media data corresponding to the task specified by the text data.

[0081] FIG. 11 is a schematic diagram illustrating an example of inference of a task execution model. For example, FIG. 11 illustrates an example in which a task execution model 130 is selected by the model selection unit 33 after a user input specifying image data 20M and text data 20T is received via a user request. As illustrated in FIG. 11, the model execution unit 35 inputs the image data 20M to the image encoder 100M of the task execution model 130 and inputs the text data 20T to the text encoder 100T of the task execution model 130. The task execution model 130 to which the image data 20M and text data 20T have been input outputs the answer "There are three oranges." In this case, the model execution unit 35 returns the output of the task execution model 130, "There are three oranges," to the client terminal 50 as a response to the user request.

[0082] 1 shows an example in which the model selection function and the model execution function are provided as a package, but the model selection function may be provided independently. In this case, the model selection unit 33 may return the output of the model selection model 100, such as identification information of the task execution model, such as its name or identifier, to the client terminal 50 in response to the user request.

[0083] <Processing flow> 12 is a flowchart showing the procedure of the inference process. As shown in FIG. 12, when the receiving unit 31 receives a user request (step S301), the model selection unit 33 inputs the media data and text data received via the user request into the model selection model 100 (step S302).

[0084] Then, the model selection unit 33 selects a task execution model that corresponds to the output of the model selection model 100 obtained as a result of step S302 from the task execution models stored in the machine learning model group storage unit 34 (step S303).

[0085] Next, the model execution unit 35 inputs the media data and text data received via the user request in step S301 into the task execution model selected in step S303 (step S304).

[0086] Thereafter, the model execution unit 35 returns the output of the task execution model obtained as a result of step S304 to the client terminal 50 as a response to the user request (step S305), and ends the process.

[0087] <Summary of Example 1> As described above, the learning device 10 according to this embodiment trains a model selection model using media data and text as explanatory variables and a task execution model selected based on the task execution results and the accuracy rate for the distribution of multiple media data including the media data as the objective variable. Therefore, a task execution model that is compatible with the task corresponding to the text specified by the user input and also with the media data specified by the user input is selected, thereby enabling high-accuracy output to be expected for both the task and the media data. Therefore, the learning device 10 according to this embodiment can select a task execution model that can be expected to provide high-accuracy output for user input.

[0088] Furthermore, the inference device 30 according to this embodiment executes processing on media data corresponding to a task specified by text data, using a task execution model trained by the learning device 10. Therefore, the inference device 30 according to this embodiment can achieve high accuracy in the output of the task execution model. Such high accuracy minimizes fine-tuning, and ultimately enables zero-shot use of multimodal AI.

[0089] <Example 2> Although the embodiments of the present disclosure have been described above, various applications are possible, and further, the present disclosure may be implemented in various different forms other than the above-described first embodiment.

[0090] <Exercise creative ability> The matters described in the first embodiment, such as the types of media data recognition tasks, and further specific examples such as the types of model selection models and task execution models and output formats, are merely examples and may be changed. Also, the order of processing in the flowchart described in the first embodiment may be changed within a consistent range.

[0091] <Model Selection Model Output> In the first embodiment, the model selection unit 33 selects one task execution model, but multiple task execution models may be selected. For example, the model selection unit may output a task execution model whose confidence level is equal to or greater than a threshold, for example, 0.8.

[0092] <Ensemble of task execution model outputs> When the model selection unit 33 selects multiple task execution models in this way, the model execution unit 35 can combine the outputs of the multiple task execution models.

[0093] Fig. 13 is a schematic diagram (1) showing an example of an ensemble. Fig. 13 shows an example in which three task execution models 131, 132, and 133 are selected by the model selection unit 33. As shown in Fig. 13, the model execution unit 35 determines the final output by majority vote of the outputs of the task execution models 131, 132, and 133. For example, in the example shown in Fig. 13, Answer A (= 1 vote) < Answer B (= 2 votes), so Answer B is determined as the final output.

[0094] Fig. 14 is a schematic diagram (2) showing an example of an ensemble. Fig. 14 also shows an example in which three task execution models 131, 132, and 133 are selected by the model selection unit 33. As shown in Fig. 14, the model execution unit 35 can also determine the final output by weighting the outputs of the task execution models 131, 132, and 133. For example, in the example shown in Fig. 14, a weight W1 is assigned to the output of task execution model 131, a weight W2 is assigned to the output of task execution model 132, and a weight W3 is assigned to the output of task execution model 133, and the sum of the weights is determined as the final output.

[0095] Here, the weights W1 to W3 can be the confidence of the labels of the task execution models 131, 132, and 133 output by the model selection model. Additionally, the weights W1 to W3 can be the accuracy rates of the task execution models 131, 132, and 133 for the cluster to which the image data specified by user input belongs, among the clusters into which the I material datasets are clustered in the training phase.

[0096] <System> The information, including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, one or more of the functional units of the generation unit 13 and training unit 15 of the learning device 10, or the reception unit 31, model selection unit 33, and model execution unit 35 of the inference device 30, may be configured as separate devices.

[0097] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown. In other words, all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Note that each configuration may also be a physical configuration.

[0098] Furthermore, each processing function performed by each device can be realized, in whole or in part, by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0099] <Hardware configuration of the learning device 10> Next, an example of the hardware configuration of the learning device 10 described in the above embodiment will be described. FIG. 15 is a diagram showing an example of the hardware configuration. As shown in FIG. 15, the learning device 10 has a communication device 10a, a storage device 10b, a memory 10c, and a processor 10d. Note that the components shown in FIG. 15 may be connected to each other via a bus or the like.

[0100] The communication device 10a is a network interface card, etc. The storage device 10b is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). For example, the storage device 10b stores programs and databases that operate the functions shown in FIG.

[0101] The processor 10d reads out a program that executes the same processing as the processing unit shown in FIG. 1 from the storage device 10b or the like and loads it into the memory 10c, thereby operating a process that executes the functions described in FIG.

[0102] Such a process realizes the same functions as the processing units of the learning device 10. For example, the processor 10d reads a program having the same functions as the generation unit 13 and the training unit 15 from the storage device 10b. Then, the processor 10d executes a process that executes the same processing as the generation unit 13 and the training unit 15.

[0103] In this way, the learning device 10 operates as an information processing device that executes a learning method by reading and executing a program. The learning device 10 can also realize functions similar to those of the above-described embodiment by reading the program from a recording medium using a media reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the learning device 10. For example, the present invention can also be applied in the same way to cases where another computer or server executes the program, or where these execute the program in cooperation with each other.

[0104] The above program can be distributed via a network such as the Internet. The above program can also be recorded on any recording medium and executed by a computer by reading it from the recording medium. For example, the recording medium can be a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), a digital versatile disk (DVD), or the like.

[0105] <Hardware Configuration of Inference Device 30> Next, an example of the hardware configuration of the inference device 30 described in the above embodiment will be described. Fig. 16 is a diagram showing an example of the hardware configuration. As shown in Fig. 16, the inference device 30 has a communication device 30a, a storage device 30b, a memory 30c, and a processor 30d. Note that the components shown in Fig. 16 may be connected to each other via a bus or the like.

[0106] The communication device 30a is a network interface card, etc. The storage device 30b is a storage device such as an HDD or SSD, etc. For example, the storage device 30b stores programs and databases that operate the functions shown in FIG.

[0107] The processor 30d reads out a program that executes the same processing as the processing unit shown in FIG. 1 from the storage device 30b or the like and loads it into the memory 30c, thereby operating the process that executes the functions described in FIG.

[0108] Such a process realizes the same functions as the processing units of the inference device 30. For example, the processor 30d reads out a program having the same functions as the reception unit 31, the model selection unit 33, the model execution unit 35, etc. from the storage device 30b, etc. Then, the processor 30d executes a process that executes the same processing as the reception unit 31, the model selection unit 33, the model execution unit 35, etc.

[0109] In this way, the inference device 30 operates as an information processing device that executes an inference method by reading and executing a program. The inference device 30 can also realize functions similar to those of the above-described embodiment by reading the program from a recording medium using a media reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the inference device 30. For example, the present invention can also be applied in the same way to cases where another computer or server executes the program, or where these execute the program in cooperation with each other.

[0110] The above program can be distributed via a network such as the Internet. The above program can also be recorded on any recording medium and executed by a computer by reading it from the recording medium. For example, the recording medium can be a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), a digital versatile disk (DVD), or the like.

[0111] The following supplementary notes are further disclosed regarding the embodiments including the above embodiments.

[0112] (Supplementary Note 1) Training data is input in which media data and text data are associated with a first machine learning model selected as a correct label from among a plurality of first machine learning models based on an execution result of a task corresponding to the text data being executed on the media data and an accuracy rate for a distribution of a plurality of media data to which the media data belongs; training parameters of a second machine learning model that outputs labels of one or more first machine learning models from the plurality of first machine learning models, using the media data and the text data of the training data as explanatory variables and the first machine learning model selected as the correct label as a target variable; A learning program that causes a computer to execute a process.

[0113] (Appendix 2) The learning program described in Appendix 1, characterized in that the input process includes a process of setting the first machine learning model among the plurality of first machine learning models whose execution result on the media data is correct as the correct answer label.

[0114] (Appendix 3) The learning program described in Appendix 1, characterized in that the input process includes a process of setting the correct label to a first machine learning model among the plurality of first machine learning models whose confidence in the label output as a result of execution of the media data is equal to or greater than a threshold.

[0115] (Appendix 4) A learning program as described in Appendix 1, characterized in that the distribution of the multiple media data corresponds to the dataset to which the media data belongs among multiple datasets used as material for generating the training data.

[0116] (Appendix 5) A learning program as described in Appendix 1, characterized in that the distribution of the multiple media data corresponds to a cluster to which the media data belongs among clusters obtained by clustering multiple datasets used as material for generating the training data by feature.

[0117] (Appendix 6) The learning program according to Appendix 1, wherein the first machine learning model and the second machine learning model are realized by a multimodal AI that processes visual information and linguistic information in an integrated manner.

[0118] (Appendix 7) The learning program according to Appendix 1, wherein the first machine learning model and the second machine learning model are realized by a multimodal AI that processes auditory information and linguistic information in an integrated manner.

[0119] (Appendix 8) Accepts the designation of media data and text data, a second machine learning model trained using training data in which media data and text data are associated with a first machine learning model selected from a plurality of first machine learning models as a correct label based on an execution result of a task corresponding to the text data being executed on the media data and an accuracy rate for a distribution of a plurality of media data to which the media data belongs, wherein the second machine learning model outputs a label of one or more first machine learning models from the plurality of first machine learning models, and selects one or more first machine learning models from the plurality of first machine learning models based on an output of the second machine learning model obtained by inputting the media data and text data whose designation has been accepted; An inference program that causes a computer to execute processing.

[0120] (Supplementary Note 9) Training data is input in which media data and text data are associated with a first machine learning model selected as a correct label from among a plurality of first machine learning models based on an execution result of a task corresponding to the text data being executed on the media data and an accuracy rate for a distribution of a plurality of media data to which the media data belongs; training parameters of a second machine learning model that outputs labels of one or more first machine learning models from the plurality of first machine learning models, using the media data and the text data of the training data as explanatory variables and the first machine learning model selected as the correct label as a target variable; A learning method characterized in that the processing is executed by a computer.

[0121] (Appendix 10) The learning method described in Appendix 9, characterized in that the input process includes a process of setting the first machine learning model among the plurality of first machine learning models for which the execution result of the media data is correct as the correct answer label.

[0122] (Appendix 11) The learning method described in Appendix 9, characterized in that the input process includes a process of setting the first machine learning model among the plurality of first machine learning models whose confidence in the label output as the execution result of the media data is equal to or greater than a threshold as the correct label.

[0123] (Appendix 12) The learning method described in Appendix 9, characterized in that the distribution of the multiple media data corresponds to the dataset to which the media data belongs among multiple datasets used as material for generating the training data.

[0124] (Appendix 13) The learning method described in Appendix 9, characterized in that the distribution of the plurality of media data corresponds to a cluster to which the media data belongs among clusters obtained by clustering, by feature, a plurality of data sets used as material for generating the training data.

[0125] (Appendix 14) The learning method according to Appendix 9, wherein the first machine learning model and the second machine learning model are realized by a multimodal AI that processes visual information and linguistic information in an integrated manner.

[0126] (Appendix 15) The learning method described in Appendix 9, wherein the first machine learning model and the second machine learning model are realized by a multimodal AI that processes auditory information and linguistic information in an integrated manner. [Explanation of symbols]

[0127] 1. Machine Learning Systems 10 Learning Device 11 Dataset group storage unit 12 Machine learning model group memory unit 13 Generation part 14 Training dataset storage 15 Training Department 30 Reasoning device 31 Reception 32 Model selection model storage unit 33 Model Selection Section 34 Machine learning model group memory unit 35 Model Execution Department

Claims

1. inputting training data in which media data and text data are associated with a first machine learning model selected as a correct label from among a plurality of first machine learning models based on an execution result of a task corresponding to the text data being executed on the media data and an accuracy rate for a distribution of a plurality of media data to which the media data belongs; training parameters of a second machine learning model that outputs labels of one or more first machine learning models from the plurality of first machine learning models, using the media data and the text data of the training data as explanatory variables and the first machine learning model selected as the correct label as a target variable; A learning program that causes a computer to execute a process.

2. The learning program according to claim 1, characterized in that the input process includes a process of setting the first machine learning model among the plurality of first machine learning models whose execution result on the media data is correct as the correct label.

3. The learning program according to claim 1, characterized in that the input process includes a process of setting the correct label to a first machine learning model among the plurality of first machine learning models whose confidence in the label output as a result of execution of the media data is equal to or greater than a threshold.

4. The learning program according to any one of claims 1 to 3, characterized in that the distribution of the plurality of media data corresponds to a dataset to which the media data belongs among a plurality of datasets used as material for generating the training data.

5. The learning program according to any one of claims 1 to 3, characterized in that the distribution of the plurality of media data corresponds to a cluster to which the media data belongs among clusters obtained by clustering, by feature, a plurality of data sets used as material for generating the training data.

6. The learning program according to claim 1 , wherein the first machine learning model and the second machine learning model are realized by a multimodal AI that processes visual information and linguistic information in an integrated manner.

7. The learning program according to claim 1 , wherein the first machine learning model and the second machine learning model are realized by a multimodal AI that processes auditory information and linguistic information in an integrated manner.

8. Accepts the designation of media data and text data, a second machine learning model trained using training data in which media data and text data are associated with a first machine learning model selected from a plurality of first machine learning models as a correct label based on an execution result of a task corresponding to the text data being executed on the media data and an accuracy rate for a distribution of a plurality of media data to which the media data belongs, wherein the second machine learning model outputs a label of one or more first machine learning models from the plurality of first machine learning models, and selects one or more first machine learning models from the plurality of first machine learning models based on an output of the second machine learning model obtained by inputting the media data and text data for which the designation has been accepted; An inference program that causes a computer to execute processing.

9. inputting training data in which media data and text data are associated with a first machine learning model selected as a correct label from among a plurality of first machine learning models based on an execution result of a task corresponding to the text data being executed on the media data and an accuracy rate for a distribution of a plurality of media data to which the media data belongs; training parameters of a second machine learning model that outputs labels of one or more first machine learning models from the plurality of first machine learning models, using the media data and the text data of the training data as explanatory variables and the first machine learning model selected as the correct label as a target variable; A learning method characterized in that the processing is executed by a computer.

Images