Deployment system for trained model, user terminal, and service providing apparatus

Abstract

To provide a deployment system for a trained model which is configured to impartially evaluate the accuracy of a model before and after relearning.SOLUTION: Fig. 9(a) shows a configuration of training data 50 in learning. Fig. 9(b) shows the training data 50 to be used in relearning of a trained model. In Fig. 9(b), hatched or dotted data is identical with the data in Fig. 9(a). In Fig. 9(b), data without hatch or dot is training data newly acquired after learning. Test data 56 in Fig. 9(a) and test data 56 in the training data 50 newly acquired after learning are used, to compare the accuracy of the model before and after relearning.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a deployment system for a trained model, a user terminal, and a service providing device. [Background technology]

[0002] For example, Patent Document 1 below describes a system for deploying a trained model generated by machine learning. In this system, a teacher dataset is prepared by adding annotation results to training data obtained from an external data source. Then, a trained model is generated by performing machine learning using the teacher dataset. Then, the trained model is deployed.

[0003] Here, multiple models are prepared as training targets. Then, the accuracy of these multiple models is evaluated using a training dataset for validation from among the training datasets. The model with the highest accuracy is then targeted for deployment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 003485 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, when new training data is accumulated after a trained model is deployed, the accuracy of the trained model is expected to improve by reflecting the new training data in the trained model. However, even if the trained model is retrained using new training data, it is not necessarily possible to create a trained model with higher accuracy than before the retraining. Therefore, if the trained model after the retraining is simply targeted for deployment, there is a risk that a downgraded trained model will be deployed. Therefore, it is desirable to fairly compare the trained model before the retraining and the trained model after the retraining to select which model to deploy. [Means for solving the problem]

[0006] Means for solving the above problems and their effects will be described below. [1] A deployment system for a trained model comprising: a storage device; and an execution device, wherein the storage device stores mapping data and teacher data, the mapping data being data that specifies a mapping determined by a trained model, and the execution device is configured to execute a teacher data acquisition process, a test data selection process, a re-learning process, a selection process, and a deployment process, wherein the teacher data acquisition process is a process of acquiring teacher data, the teacher data being data that specifies values ​​of input variables and an output of the mapping, the test data selection process is a process of selecting a portion of the teacher data to use as test data, the relearning process includes a process of relearning the mapping using data other than the test data as input without referring to the test data, the selection process is a process of selecting which of the mappings before and after the relearning is to be adopted by comparing the accuracy of the re-learned mapping using the test data with the accuracy of the mapping before the relearning, and the deployment process includes a process of deploying the mapping after the relearning if the mapping after the relearning is selected by the selection process.

[0007] In the above configuration, the accuracy of the mappings before and after the new re-learning is evaluated using test data that is not used in the re-learning of either of the pair of mappings to be compared. Therefore, the accuracy of the pair of mappings can be fairly evaluated. Therefore, a trained model with good accuracy can be deployed. In particular, since the execution device executes each of the above processes, the deployment of a new trained model by re-learning can be automated.

[0008] [2] The deployment system for a trained model described in claim 1, wherein the re-learning process includes a process of dividing data of the teacher data other than the test data into verification data and training data, learning the mapping using the training data, and evaluating the learning of the mapping using the training data using the verification data.

[0009] In the above configuration, in the relearning of any pair of mappings to be compared, data that is neither the training data nor the validation data can be used as test data. [3] A deployment system for a trained model described in 1 or 2 above, wherein the execution device is configured to execute a teacher data storage process, the teacher data storage process is a process of storing the teacher data acquired by the teacher data acquisition process, the re-learning process includes, when re-learning is performed newly in a state where the re-learning has already been performed, a process of performing new re-learning using as input at least data other than the test data among the teacher data newly acquired by the teacher data acquisition process after the past re-learning, and the selection process is a process of selecting which of the mappings before or after the re-learning to adopt using as input the test data among the teacher data acquired by the teacher data acquisition process before the past re-learning and the test data among the teacher data newly acquired by the teacher data acquisition process after the past re-learning.

[0010] In the above configuration, the larger the amount of data used to train the model, the larger the amount of test data used to compare the accuracy of models can be. [4] A deployment system for a trained model described in claim 3, wherein the re-learning process includes, in a case where new re-learning is performed after the re-learning has already been performed, a process of performing new re-learning using as input data other than the test data among the teacher data newly acquired by the teacher data acquisition process after the past re-learning, and data other than the test data among the teacher data acquired by the teacher data acquisition process before the past re-learning.

[0011] In the above configuration, new relearning can be performed using teacher data acquired before the new relearning. [5] The execution device is configured to execute preprocessing, and the preprocessing is a process of aligning data that is an input to the mapping among the training data with the input to the mapping.

[0012] In the above configuration, the execution device executes the preprocessing. This simplifies the process of a person providing training data to the system. Therefore, even a user who does not know about machine learning or the mechanism of the target model can provide training data.

[0013] [6] A deployment system for a trained model according to any one of 1 to 5 above, wherein the storage device stores model defining data, the model defining data being data defining a plurality of models having different outputs from each other, the execution device is configured to execute an instruction receiving process, a model selection process, and a learning process, the instruction receiving process being a process of receiving instructions for an input of the mapping and an output of the mapping, the output instructions including at least one of an instruction for the number of dimensions of the output of the mapping and an instruction as to whether the output variable of the mapping is a continuous variable or a label variable, the model selection process being a process of selecting the model that matches the instruction received by the instruction receiving process from among the plurality of models defined by the model defining data, and the learning process including a process of updating parameters of the model selected by the model selection process using data from the teacher data that was not selected as the test data in order to generate the mapping data.

[0014] In the above configuration, the execution device selects a model that matches the input and output instructions from the user. Then, the parameters of the selected model are updated using data that was not selected as test data from the teacher data. Therefore, a user who is not familiar with machine learning can automatically generate a desired trained model by simply specifying what kind of output is desired for the input.

[0015] Furthermore, the data selected as test data is not used for training. Therefore, a portion of the training data already acquired during training can be used as test data, allowing for a fair comparison of accuracy before and after re-training.

[0016] [7] A deployment system for a trained model described in any one of 1 to 6 above, wherein the storage device stores model definition data, the model definition data being data defining a plurality of models each having a different output, the execution device is configured to execute a change acceptance process, a model change process, and a change learning process, the change acceptance process being a process of accepting a change instruction for a model defining the mapping, the change instruction including an instruction to change the output of the model, the model change process being a process of changing the model defining the mapping to a model that conforms to the instruction accepted by the change acceptance process among the plurality of models defined by the model definition data, and the change learning process including a process of updating parameters of a model in accordance with the change instruction using data from the teacher data that was not selected as test data.

[0017] In the above configuration, when a user wants to change the output, the execution device selects a model that matches the user's desire. Then, the parameters of the selected model are updated using data that was not selected as test data from among the teacher data. Therefore, a user who is not familiar with machine learning can automatically change to the desired trained model by simply specifying what kind of output he or she wants.

[0018] Furthermore, the data selected as test data is not used for training. Therefore, a portion of the training data already acquired during training can be used as test data, allowing for a fair comparison of accuracy before and after re-training after the change.

[0019] [8] The execution device is configured to execute a judgment process, which is a process for judging whether or not the change learning process can use the teacher data acquired by the teacher data acquisition process before the change instruction, and the change learning process includes a process for updating parameters of the model using both data of the teacher data acquired by the teacher data acquisition process before the change instruction that was not selected as the test data and data of the teacher data acquired by the teacher data acquisition process after the change instruction that was not selected as the test data if the judgment process judges that the teacher data can be used.

[0020] In the above configuration, by determining whether or not the training data acquired before the change instruction can be used for the change learning process, the changed model can be learned using appropriate training data.

[0021] Furthermore, the data selected as test data is not used for training the modified model. Therefore, a portion of the training data already acquired during training can be used as test data, allowing a fair comparison of the accuracy before and after re-training after the modification.

[0022] [9] A deployment system for a trained model described in any one of 1 to 8 above, wherein the execution device comprises a first execution device and a second execution device, the first execution device is configured to execute the teacher data acquisition process and the teacher data transmission process, the second execution device is configured to execute the teacher data reception process, the re-learning process, and the selection process, the teacher data transmission process is a process of transmitting the teacher data acquired by the teacher data acquisition process, and the teacher data reception process is a process of receiving the teacher data transmitted by the teacher data transmission process.

[0023] In the above configuration, as long as the first execution device is in an area where communication with the second execution device is possible, the second execution device can execute re-learning processing using the teacher data acquired by the first execution device, no matter where the first execution device is located.

[0024]

[10] A deployment system for a trained model as described in claim 9, wherein the first execution device is configured to execute an input acceptance process, an input transmission process, a result reception process, and a result notification process, and the second execution device is configured to execute an input reception process, a calculation process, and a result transmission process, wherein the input acceptance process is a process of accepting values ​​of input variables of the mapping, the input transmission process is a process of transmitting values ​​of the input variables accepted by the input acceptance process, the input reception process is a process of receiving values ​​of the input variables transmitted by the input transmission process, the calculation process is a process of calculating an output of the mapping by inputting values ​​of the input variables received by the input reception process to the mapping, the result transmission process is a process of transmitting the output of the mapping, the result reception process is a process of receiving the output transmitted by the result transmission process, and the result notification process is a process of notifying the output received by the result reception process.

[0025] In the above configuration, no matter where the first execution device is located, as long as it is in an area where communication with the second execution device is possible, the first execution device can notify the result calculated by the second execution device in response to an input accepted by the first execution device.

[0026]

[11] A user terminal comprising the first execution device of the trained model deployment system described in 9 or 10 above.

[12] A service providing device comprising the second execution device and the storage device of the trained model deployment system described in 9 or 10 above. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a block diagram showing a configuration of a deployment system for a trained model according to an embodiment. [Diagram 2]11(a) and 11(b) are flowcharts showing the procedure of processing executed by the system according to the embodiment. [Diagram 3] 11(a) and 11(b) are flowcharts showing the procedure of processing executed by the system according to the embodiment. [Figure 4] 13(a) and 13(b) are diagrams illustrating an example of generation of teacher data according to the embodiment. [Diagram 5] 13(a) and 13(b) are diagrams illustrating an example of generation of teacher data according to the embodiment. [Figure 6] 11 is a flowchart showing a procedure of a process executed by a service providing device according to the embodiment. [Figure 7] 11(a) and 11(b) are flowcharts showing the procedure of processing executed by the system according to the embodiment. [Figure 8] 11(a) and 11(b) are flowcharts showing the procedure of processing executed by the system according to the embodiment. [Figure 9] 13(a) and (b) are diagrams illustrating an example of how training data is divided in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, an embodiment will be described with reference to the drawings. "Prerequisite configuration" The trained model deployment system shown in Figure 1 is a system that automatically performs training, deployment, re-training, and redeployment of trained models.

[0029] The user terminal 10 is a terminal owned by a user who uses the services of this system. It is assumed that the user in this embodiment does not have the knowledge to develop machine learning on his / her own. It is assumed that the user wants to use a trained model that solves a specific problem. The user terminal 10 may be, for example, a multi-function mobile terminal such as a multi-function mobile phone.

[0030] The user terminal 10 includes an input / output device 10a. The input / output device 10a functions as both an input device through which a user performs input operations and an output device that provides information to the user. The input / output device 10a is, for example, a display overlaid with a touch panel.

[0031] The user terminal 10 includes a PU 12, a storage device 14, and a communication device 16. The PU 12 is a software processing device including at least one of a CPU, a GPU, and a TPU. The storage device 14 stores an AI generation program 14a and an AI utilization program 14b. The communication device 16 is a device that communicates with the service providing device 30 via the network 20.

[0032] The service providing device 30 is a device that automatically creates a trained model according to the desires of a user who has no knowledge of machine learning, and enables the user to utilize the trained model. The service providing device 30 includes a PU 32, a storage device 34, and a communication device 36. The PU 12 is a software processing device that includes at least one of a CPU, a GPU, and a TPU. The storage device 34 stores a learning program 34a, model definition data 34b, mapping data 34c, a teacher data group 34d, and an AI providing program 34e. The communication device 36 is a device that communicates with the communication device 16 via the network 20.

[0033] "Determining the specifications of the trained model" FIG. 2 shows the procedure for the process of determining the specifications of a trained model. The process shown in FIG. 2(a) is realized by the PU 12 repeatedly executing the AI ​​generation program 14a at a predetermined cycle, for example. FIG. 2(b) is realized by the PU 32 repeatedly executing the learning program 34a at a predetermined cycle, for example. In the following, the step number of each process is represented by a number preceded by "S."

[0034] In the series of processes shown in FIG. 2(a), the PU 12 first receives an instruction regarding the specifications of the trained model (S10). Prior to this process, the PU 12 may, for example, display on the input / output device 10a a screen inquiring about the relationship between the desired input and output. In response to this, the user specifies the input and output according to the problem he or she wants to solve by performing an input operation on the input / output device 10a. For example, if the user wants to know the position of an apple in a color image, the input is a three-dimensional color image and the output is a variable value indicating the position of the apple. For example, if the user wants to know the sweetness of an apple in a color image, the input is a three-dimensional color image and the output is a variable indicating the sweetness of the apple. For example, if the user wants to know the sweetness and sourness of an apple in a color image, the input is a three-dimensional color image and the output is a variable indicating the sweetness and sourness of the apple. Note that the input of the trained model is not limited to a color image. The input of the trained model may be, for example, a monochrome image. Furthermore, the input of the trained model is not limited to images, but may be, for example, text data, audio data, etc.

[0035] It is desirable for the PU 12 to display on the input / output device 10a not only the number of outputs but also guidance information that allows the user to select whether the outputs are continuous variables or label variables. In the above example, the position variable of the apple is a continuous variable. On the other hand, the sweetness and sourness of the apple can be expressed as both a label variable and a continuous variable. It is up to the user to choose which one to use.

[0036] When the PU 12 receives an instruction regarding the model specifications, it transmits the received instruction by operating the communication device 16 (S12). In response to this, as shown in FIG. 2(a), the PU 32 of the service providing device 30 receives the transmitted instruction (S20). Then, the PU 32 selects a model that best suits the received instruction from among a plurality of models defined by the model defining data 34b (S22). The plurality of models defined by the model defining data 34b include models with various input dimensional numbers and output dimensional numbers. Furthermore, the plurality of models may include a model whose output is a label variable, a model whose output is a continuous variable, and a model whose output is both a label variable and a continuous variable. An example of the models with different outputs is a parametric model. An example of the models with different outputs is a neural network with the same structure except for the final layer.

[0037] For example, if an output with only one label variable is indicated, the PU 32 selects a neural network with only one label variable and no continuous variable outputs. On the other hand, the PU 12 of the user terminal 10 judges whether or not an instruction on the learning method has been inputted as shown in FIG. 2(a) (S14). This may be performed by the PU 12 displaying information for selecting the update timing of the trained model on the input / output device 10a. Specifically, the PU 12 may display information on the input / output device 10a that allows the update timing to be selected by day of the week or by the number of days that specifies the cycle. When the user inputs an instruction on the learning method (S14: YES), the PU 12 transmits the instruction content by operating the communication device 16 (S16). Note that the PU 12 temporarily ends the series of processes shown in FIG. 2(a) when completing the process of S16 or when making a negative judgment in the process of S14.

[0038] On the other hand, as shown in Fig. 2(b), when the PU 32 of the service providing device 30 receives an instruction on how to learn (S24: YES), the PU 32 stores the instruction in the storage device 34 (S26). On the other hand, when the PU 32 does not receive an instruction on how to learn (S24: NO), the PU 32 sets the learning conditions as default conditions (S28).

[0039] When the PU 32 completes the processes of S26 and S28, the PU 32 temporarily ends the series of processes shown in FIG. 2(b). "Processing for the collection of teacher data" The procedure for the process of collecting teacher data is shown in Fig. 3. The process shown in Fig. 3(a) is realized by the PU 12 repeatedly executing the AI ​​generation program 14a at a predetermined cycle, for example. The process shown in Fig. 3(b) is realized by the PU 32 repeatedly executing the learning program 34a at a predetermined cycle, for example.

[0040] 3(a), the PU 12 of the user terminal 10 first determines whether or not the user has input teacher data by operating the input / output device 10a (S30). The teacher data is generated by the user adding annotations to input data.

[0041] A method for generating training data is illustrated in Figure 4. Figure 4 shows an example in which a user is looking for a trained model that, when given image data, points to an image of an apple in the image shown by the image data.

[0042] Fig. 4(a) shows an image 40 captured by a user. In the example shown in Fig. 4(a), the image 40 includes an apple 42. As shown in Fig. 4(b), the user performs annotation by surrounding the apple 42 in the image 40 with a marking 44. This generates training data including image data and annotation data.

[0043] FIG. 5 shows another example of a method for generating training data. FIG. 5 shows an example in which a user is looking for a trained model that outputs the sweetness of an apple when image data showing an image including an apple is given. FIG. 5(a) shows an image 40 taken by a user. The image 40 shown in FIG. 5(a) includes an apple 42. The user tastes the apple himself, and then inputs the evaluation result of the sweetness of the apple via the input / output device 10a. FIG. 5(b) shows an example in which the sweetness is the second of three levels, "1, 2, 3". This corresponds to the user selecting a label variable as an output in the process of S10. The training data is composed of data showing the image 40 and data 46 showing the sweetness.

[0044] 3, when the PU 12 determines that the teacher data has been input (S30: YES), the PU 12 transmits the teacher data (S32) by operating the communication device 16. When the PU 12 completes the process of S32 or when the process of S30 is determined to be negative, the PU 12 temporarily ends the series of processes shown in FIG. 3(a).

[0045] On the other hand, as shown in FIG. 3(b), when the PU 32 of the service providing device 30 receives the teacher data (S40: YES), the PU 32 performs preprocessing on the received teacher data (S42). The preprocessing includes a process of matching data of the teacher data that is to be an input of the trained model with the input of the model selected by the process of S22. For example, when the input is image data, if the number of pixels of the image data included in the received teacher data is greater than the number of pixels of the image data of the model, the PU 32 executes a process of thinning out the pixels of the image data included in the teacher data. Also, for example, when the input is image data, if the number of pixels of the image data included in the received teacher data is less than the number of pixels of the image data of the model, the PU 32 executes a process of increasing the number of pixels of the image data included in the teacher data by interpolation.

[0046] Next, the PU 32 stores the teacher data that has been subjected to the process of S42 in the storage device 34 (S44). By repeating the process of S44, the teacher data group 34d described above is generated.

[0047] Then, the PU 32 divides the teacher data group 34d into test data and non-test data (S46). As an example, this process may be a process in which the PU 32 assigns a label to each piece of teacher data indicating whether it is test data or not.

[0048] When the process of S46 is completed or when a negative determination is made in the process of S40, the PU 32 temporarily ends the series of processes shown in FIG. "Model learning" A procedure for processing related to model learning is shown in Fig. 6. The processing shown in Fig. 6 is realized by the PU 32 repeatedly executing the learning program 34a, for example, at a predetermined interval.

[0049] In the series of processes shown in Fig. 6, the PU 32 first judges whether or not the learning condition set by the process of S26 or the process of S28 is satisfied (S50). When the PU 32 judges that the learning condition is satisfied (S50: YES), the PU 32 divides the teacher data that was not selected as the test data in the process of S46 into training data and verification data (S52). Next, the PU 32 reads the model selected in the process of S22 (S54). Then, the PU 32 updates the parameter values ​​of the model using the training data (S56). In detail, the PU 32 updates the parameter values ​​so that the difference between the output and the annotation data when input data of the training data is input to the model is reduced.

[0050] The PU 32 then uses the validation data to evaluate the accuracy of the model with the updated parameter values ​​(S58). The process of S58 may be cross-validation. Then, the PU 60 determines whether the learning is completed based on the result of the process of S58 (S60). This process may be a process in which the PU 32 determines that the learning is completed when the time point at which the accuracy is maximized is detected by accuracy evaluation using the verification data, for example.

[0051] When the PU 32 determines that the learning is not yet complete (S60: NO), the PU 32 returns to the process of S56. On the other hand, when the PU 32 determines that the learning is complete (S60: YES), the PU 32 determines whether or not an old model exists (S62). When the current process of S56 to S60 uses teacher data newly acquired after the process of S68 described below is completed, the PU 32 determines that an old model exists. In this case, the model determined to be positive by the process of S60 becomes the model after re-learning.

[0052] When it is determined that an old model exists (S62: YES), the PU 32 uses the test data to compare the accuracy of the old model with the accuracy of the model after the current training (S64). That is, the PU 32 evaluates the output when input data of the test data is input to each model.

[0053] When the PU 32 determines that the accuracy of the newly learned model is higher than that of the old model (S66: YES), it deploys the newly learned model (S68). The data that defines the mapping defined by this newly learned model is the above-mentioned mapping data 34c. This means that the newly learned model is used in the service provided to deal with the user's problem instead of the old model.

[0054] On the other hand, even if the PU 62 makes a negative determination in the process of S62, the PU 62 proceeds to the process of S68. In this case, the process of S68 is a process in which the trained model having the specifications instructed by the user in the process of S10 is deployed for the first time.

[0055] When the process of S68 is completed or when a negative determination is made in the processes of S50 and S66, the PU 32 temporarily ends the series of processes shown in FIG. "Handling Model Changes" The procedure for processing a model change is shown in Fig. 7. The process shown in Fig. 7(a) is realized by the PU 12 repeatedly executing the AI ​​generation program 14a at a predetermined cycle, for example. The process shown in Fig. 7(b) is realized by the PU 32 repeatedly executing the learning program 34a at a predetermined cycle, for example.

[0056] In the series of processes shown in FIG. 7(a), the PU 12 of the user terminal 10 first judges whether or not the user has instructed to change the model by inputting an input / output device 10a (S70). Here, the instruction to change is, for example, an instruction to change the number of dimensions of the output. That is, an instruction to change from a mapping that takes an image of an apple as input and outputs sweetness to a mapping that takes an image of an apple as input and outputs sweetness and sourness corresponds to this. There is also an instruction to change the number of dimensions of the input, for example. For example, an instruction to change from a mapping that takes date of birth, height, weight, daily calorie intake, and daily amount of exercise as input and outputs risk of diabetes to an input that further includes sleep time corresponds to this.

[0057] When PU 12 determines that a change instruction has been given (S70: YES), it checks whether or not the user has given an instruction to not use the teacher data before the change instruction to learn the changed model by inputting an input operation on input / output device 10a (S72).

[0058] When the process of S72 is completed, the PU 12 operates the communication device 16 to transmit an instruction corresponding to the process of S72 and the process of S74 (S74). That is, the PU 72 transmits information indicating an instruction on the contents of the change. The PU 72 also transmits information indicating an instruction on whether or not to use the pre-change teacher data for learning the post-change model.

[0059] When the process of S74 is completed or when a negative determination is made in the process of S70, the PU 32 temporarily ends the series of processes shown in FIG. 7A. On the other hand, as shown in Fig. 7(b), when the PU 32 of the service providing device 30 receives the instruction (S80: YES), it selects a model according to the change instruction from among the multiple models defined by the model definition data 34b (S82). Next, the PU 32 determines whether or not an instruction has been given to not use the teacher data acquired before the change instruction for learning the changed model (S84). If the PU 32 determines that an instruction not to use has been given (S84: YES), it deletes the teacher data group 34d stored in the storage device 34 (S86).

[0060] When the process of S86 is completed or when a negative determination is made in the process of S84, the PU 32 temporarily ends the series of processes shown in FIG. 7B. After the process of S82 is performed, the modified model is trained by the process shown in Fig. 6. If the process of S86 is not performed, the training data acquired before the change is also used for training the modified model. For example, if a model that takes image data as input and outputs the position of an apple is changed to a model that takes image data as input and outputs the positions of apples and oranges, it is highly likely that the training data before the change does not include data on the position of oranges. In that case, it is assumed that oranges do not exist in the image data before the change.

[0061] "Using a pre-trained model" The procedure of the process for using the trained model is shown in Fig. 8. The process shown in Fig. 8(a) is realized by the PU 12 repeatedly executing the AI-use program 14b at a predetermined cycle, for example. The process shown in Fig. 8(b) is realized by the PU 32 repeatedly executing the AI-providing program 34e at a predetermined cycle, for example.

[0062] 8(a), the PU 12 of the user terminal 10 first judges whether data indicating the input of the mapping defined by the mapping data 34c has been input by an input operation on the input / output device 10a (S90). If the PU 12 judges that data indicating the input of the mapping has been input (S90: YES), the PU 12 operates the communication device 16 to transmit the data indicating the input (S92).

[0063] On the other hand, as shown in FIG. 8(b), the PU 32 of the service providing device 30 receives data indicating an input (S100). Then, the PU 32 reads out the mapping data 34c and inputs the received input data into the mapping defined by the mapping data 34c (S102). As a result, the PU 32 calculates the output of the mapping. Then, the PU 32 transmits data related to the output by operating the communication device 36 (S104). When the PU 32 completes the process of S104, it temporarily ends the series of processes shown in FIG. 8(b).

[0064] 8(a), the PU 12 of the user terminal 10 receives the transmitted data regarding the output (S94). The PU 12 then operates the input / output device 10a to display visual information indicating the output (S96).

[0065] When the process of S96 is completed or when a negative determination is made in the process of S90, the PU 12 temporarily ends the series of processes shown in FIG. 8(a). "Actions and Effects of the Present Embodiment" When the PU 32 receives the teacher data provided by the user, the PU 32 selects a part of the teacher data as test data, and divides the teacher data other than the test data into training data and verification data.

[0066] 9(a) shows the configuration of teacher data 50 at the time of the initial learning. The teacher data 50 is divided into training data 52, verification data 54, and test data 56. Fig. 9(b) shows training data 50 used when relearning a model after training. In the example shown in Fig. 9(b), new training data has been added to the data shown in Fig. 9(a). In Fig. 9(b), data with hatching or dots indicates that the data is the same as the data in Fig. 9(a). In Fig. 9(b), data with neither hatching nor dots indicates that the data is training data newly acquired after training.

[0067] In the re-learning, the PU 32 sets a part of the training data 52 shown in Fig. 9(a), a part of the verification data 54 shown in Fig. 9(a), and a part of the teacher data newly acquired after the learning as the training data 52. In the re-learning, the PU 32 sets a part of the training data 52 shown in Fig. 9(a), a part of the verification data shown in Fig. 9(a), and a part of the teacher data newly acquired after the learning as the verification data 54.

[0068] Then, the PU 32 compares the accuracy of the model before and after re-learning, using the test data 56 shown in FIG. 9(a) and the test data 56 of the teacher data 50 newly acquired after learning.

[0069] 9(a) and the test data 56 in the teacher data 50 newly acquired after learning are not used in learning any of the models before and after re-learning. Therefore, it is possible to fairly determine which model has higher accuracy before or after re-learning.

[0070] 9, for the sake of convenience, the amount of teacher data 50 acquired up to the first learning is shown to be equal to the amount of teacher data 50 acquired from the completion of the first learning until re-learning. However, in reality, for example, the amount of teacher data 50 acquired from the completion of the first learning until re-learning may be less than the amount of teacher data 50 acquired up to the first learning.

[0071] According to the present embodiment described above, the following actions and effects can be obtained. (1) In this re-learning, the PU32 newly learned the model using the teacher data used in the learning before the re-learning, instead of updating the already-learned model parameters. This makes it easier to improve the reliability of the learning compared to updating the already-learned model parameters. That is, for example, if the number of teacher data acquired from the previous learning to the current learning is insufficient, there is a risk that the completion of the learning cannot be appropriately determined if the already-learned model parameters are updated using only the teacher data.

[0072] <Correspondence> The correspondence between the matters in the above embodiment and the matters described in the above "Means for solving the problem" column is as follows. In the following, the correspondence is shown for each number of the solving means described in the "Means for solving the problem" column. [1] The execution device corresponds to the PU 12, 32 and the storage device 14, 34. The storage device corresponds to the storage device 34. The teacher data acquisition process corresponds to the process of S30. The test data selection process corresponds to the process of S46. The re-learning process corresponds to the processes of S52 to S60 in the case where learning has been performed once. The selection process corresponds to the processes of S64, S66. The deployment process corresponds to the process of S68. [2] Corresponds to the process of S52. [3, 4] The teacher data storage process corresponds to the process of S44. The setting of the test data corresponds to FIG. 9. [5] The pre-processing corresponds to the process of S42. [6] The designation reception process corresponds to the process of S10. The model selection process corresponds to the process of S22. The learning process corresponds to the processes of S52 to S60 in a state where the trained model is not deployed. [7] The change acceptance process corresponds to the process of S70. The model change process corresponds to the process of S82. The change learning process corresponds to the processes of S52 to S60, which are executed immediately after the process of S82. [8] The determination process corresponds to the process of S84. [9] The first execution device corresponds to the PU 12 and the storage device 14. The second execution device corresponds to the PU 32 and the storage device 34. The teacher data transmission process corresponds to the process of S32. The teacher data reception process corresponds to the process of S40.

[10] The input acceptance process corresponds to the process of S90. The input transmission process corresponds to the process of S92. The result reception process corresponds to the process of S94. The result notification process corresponds to the process of S96. The input reception process corresponds to the process of S100. The calculation process corresponds to the process of S102. The result transmission process corresponds to the process of S104.

[11] It corresponds to the user terminal 10.

[12] Corresponding to the service providing device 30.

[0073] <Other embodiments> This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that no technical contradiction occurs.

[0074] "About the learning process" In the above embodiment, the process of S22 selects one of a plurality of models, but this is not limiting. For example, a plurality of models that match the instructions of the models defined by the model definition data 34b may be selected. In this case, the PU 32 may adopt the model with the highest accuracy evaluated by the verification data, for example.

[0075] "About the Re-learning Process" In the above embodiment, in the re-learning process, re-learning is performed using all data that was not selected as test data among all teacher data accumulated up to that point, but this is not limited to the above. Re-learning may be performed using only data that was not selected as test data among teacher data newly acquired since the process of S68 was last executed. In this case, it is preferable that the re-learning is a process of updating the parameters of the deployed model instead of updating the parameters of the model from the initial state. Also, for example, re-learning may be performed using only data that was not selected as test data among teacher data up to the most recent N(>1) times before the process of S68 was executed. In this case, teacher data acquired before the most recent N(>1) times may be deleted.

[0076] "About the test data selection process" In the above embodiment, the PU 32 assigns a label to each piece of teacher data indicating whether it is test data or not, but this is not limited to the above. For example, by defining an area for storing test data and an area for storing data other than test data in the storage device 34, it may be possible to define whether the teacher data is test data or not depending on the area in which the teacher data is stored.

[0077] "About selection processing" In the above embodiment, the accuracy of both models before and after re-learning is evaluated using all data selected as test data from all teacher data accumulated up to that point, but this is not limited to the above. For example, the accuracy of both models before and after re-learning may be evaluated using only data selected as test data from teacher data up to the most recent N (>1) times before the processing of S68 was executed. In this case, data selected as test data from teacher data acquired before the most recent N (>1) times may be deleted.

[0078] "About the judgment process" In FIG. 7, whether past teacher data can be used for learning the changed model when the model is changed is determined based on an instruction from a user, but this is not limited to the above. For example, the PU32 may determine using teacher data after the change and teacher data before the change. When this process is applied to a process for detecting different types of objects, for example, it may be realized as follows. That is, if the model before the change is a model for detecting apples and the model after the change is a model for detecting apples and oranges, when the teacher data before the change is used, the output related to oranges may not be evaluated. That is, in this case, the PU32 determines that the teacher data before the change can be used for learning after the change. Also, for example, if the number of input items of the changed model is smaller than that of the changed model, the PU32 may determine that the teacher data before the change can be used for learning after the change. On the other hand, for example, if the number of input items of the changed model is larger than that of the changed model, the PU32 may determine that the teacher data before the change cannot be used for learning after the change.

[0079] "About trained models for specifying mappings" The trained model is not limited to a parametric model. For example, it may be a nonparametric model. If the nonparametric model is a discrimination model, for example, a support vector machine can be used. If the nonparametric model is a regression model, for example, a support vector regression model can be used.

[0080] "About the execution device" The execution device is not limited to one that executes software processing. For example, it may be equipped with a dedicated hardware circuit such as an ASIC that executes at least a part of the processing executed in the above embodiment. That is, the execution device may include a processing circuit having any of the following configurations (a) to (c). (a) A processing circuit having a processing device that executes all of the above processing according to a program, and a program storage device such as a memory device that stores the program. (b) A processing circuit having a processing device and a program storage device that executes part of the above processing according to a program, and a dedicated hardware circuit that executes the remaining processing. (c) A processing circuit having a dedicated hardware circuit that executes all of the above processing. Here, there may be multiple software execution devices equipped with a processing device and a program storage device. Also, there may be multiple dedicated hardware circuits.

[0081] "About the deployment system" It is not essential that the deployment system is composed of the user terminal 10 and the service providing device 30. For example, the device that executes the processes shown in Fig. 2(b), Fig. 3(b), Fig. 6, and Fig. 7(b) and the device that executes the process shown in Fig. 8(b) may be separate devices.

[0082] The deployment system does not necessarily have to be composed of multiple devices with different owners. For example, all of the processes illustrated in the above embodiment, except for the process of transmitting and receiving data via the network 20, may be executed by the user terminal 10. [Explanation of symbols]

[0083] 10...User terminal 20…Network 30...Service provision device 40...Image 42...Apple 44…Marking

Claims

1. A storage device and an execution device, The storage device stores mapping data and teacher data, The mapping data is data that specifies a mapping determined by a trained model, The execution device is configured to execute a teacher data acquisition process, a test data selection process, a re-learning process, a selection process, and a deployment process; The teacher data acquisition process is a process of acquiring teacher data, The teaching data is data that specifies values ​​of input variables and outputs of the mapping, The test data selection process is a process of selecting a part of the teacher data as test data, the re-learning process includes a process of re-learning the mapping by using data other than the test data among the teacher data as an input without referring to the test data, the selection process is a process of selecting which of the mappings before and after the relearning is to be adopted by comparing an accuracy of the re-learned mapping with an accuracy of the mapping before the relearning using the test data; A system for deploying a trained model, wherein the deployment process includes a process of deploying the mapping after re-training when the mapping after re-training is selected by the selection process.

2. 2. The system for deploying a trained model according to claim 1, wherein the re-learning process includes a process of dividing data other than the test data among the teacher data into verification data and training data, learning the mapping using the training data, and evaluating the learning of the mapping using the training data using the verification data.

3. The execution device is configured to execute a teacher data storage process, The teacher data storage process is a process of storing the teacher data acquired by the teacher data acquisition process, The re-learning process includes, in a case where new re-learning is performed in a state where the re-learning has already been performed, a process of performing new re-learning using data other than the test data among the teacher data newly acquired by the teacher data acquisition process after at least the past re-learning as an input, The deployment system for a trained model as described in claim 1, wherein the selection process is a process of selecting which of the mappings before or after re-learning to adopt by using as input the test data of the teacher data acquired by the teacher data acquisition process before the past re-learning and the test data of the teacher data newly acquired by the teacher data acquisition process after the past re-learning.

4. The deployment system for a trained model according to claim 3, wherein the re-learning process includes, when new re-learning is performed after the re-learning has already been performed, a process of performing new re-learning using as input data other than the test data among the teacher data newly acquired by the teacher data acquisition process after the past re-learning, and data other than the test data among the teacher data acquired by the teacher data acquisition process before the past re-learning.

5. the execution unit is configured to perform pre-processing; The system for deploying a trained model according to claim 1 , wherein the preprocessing is a process of aligning data among the training data that is to be input to the mapping with the input to the mapping.

6. The storage device stores model defining data; The model defining data is data defining a plurality of models having different outputs, The execution device is configured to execute an instruction receiving process, a model selection process, and a learning process; the instruction receiving process is a process of receiving an instruction for inputting the mapping and outputting the mapping, The output indication includes at least one of two: an indication of the number of dimensions of the output of the mapping; and an indication of whether the output variable of the mapping is a continuous variable or a label variable; the model selection process is a process of selecting a model that matches the instruction received by the instruction receiving process from among a plurality of models defined by the model definition data, 2. The trained model deployment system of claim 1, wherein the learning process includes a process of updating parameters of the model selected by the model selection process using data from the teacher data that was not selected as the test data in order to generate the mapping data.

7. The storage device stores model defining data; The model defining data is data defining a plurality of models having different outputs, The execution device is configured to execute a change acceptance process, a model change process, and a change learning process; the change acceptance process is a process of accepting an instruction to change a model that defines the mapping, the modification instructions include instructions to modify an output of the model; the model change process is a process of changing the model that defines the mapping to a model that conforms to an instruction accepted by the change acceptance process among a plurality of models defined by the model definition data, The trained model deployment system of claim 1 , wherein the change learning process includes a process of updating parameters of the model in accordance with the change instruction using data from the training data that was not selected as the test data.

8. the execution device is configured to execute a determination process; The determination process is a process for determining whether or not the change learning process can use the teacher data acquired by the teacher data acquisition process before the change instruction, The trained model deployment system of claim 7, wherein the change learning process includes, when it is determined by the judgment process that the model can be used, a process of updating parameters of the model using both data of the teacher data acquired by the teacher data acquisition process before the change instruction that was not selected as the test data and data of the teacher data acquired by the teacher data acquisition process after the change instruction that was not selected as the test data.

9. the execution device comprises a first execution device and a second execution device; The first execution device is configured to execute the teacher data acquisition process and the teacher data transmission process, The second execution device is configured to execute a teacher data receiving process, the relearning process, and the selection process; The teacher data transmission process is a process of transmitting the teacher data acquired by the teacher data acquisition process, The trained model deployment system according to claim 1 , wherein the teacher data reception process is a process of receiving the teacher data transmitted by the teacher data transmission process.

10. the first execution unit is configured to execute an input reception process, an input transmission process, a result reception process, and a result notification process; The second execution unit is configured to execute an input receiving process, a calculation process, and a result sending process; the input reception process is a process of receiving values ​​of input variables of the mapping, the input transmission process is a process of transmitting values ​​of the input variables accepted by the input acceptance process, the input receiving process is a process of receiving the value of the input variable transmitted by the input transmitting process, the calculation process is a process of calculating an output of the mapping by inputting values ​​of the input variables received by the input reception process to the mapping; the result transmission process is a process of transmitting an output of the mapping; the result receiving process is a process of receiving the output transmitted by the result transmitting process, The trained model deployment system according to claim 9 , wherein the result notification process is a process of notifying the output received by the result reception process.

11. A user terminal comprising the first execution device of the trained model deployment system according to claim 9.

12. A service providing device comprising the second execution device and the storage device of the trained model deployment system according to claim 9.

Images