Service provision system, method, program, service provision device, method, and program

Abstract

To improve prediction accuracy using an appropriate model.SOLUTION: A service provision system according to an embodiment of the present invention includes a service provision device and a user terminal. The service provision device includes: a model storage unit configured to store a first model learned using feature quantities of explanatory variables and a second model learned using invariant feature quantities, which are feature quantities of the explanatory variables that do not receive new values during prediction; a learned feature quantity storage unit configured to store learned feature quantities for feature quantities other than the invariant feature quantities; and a model switching execution unit configured to execute predictions using the first model when the learned feature quantities are input and the second model when a value other than the learned feature quantities is input, and output predicted values. The user terminal includes display means for displaying the predicted values received from the service provision device.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a service providing system, a method, a program, a service providing device, a method, and a program. [Background technology]

[0002] Artificial intelligence (AI) has been increasingly used in various fields. Technology that prepares and applies an AI model (hereinafter simply referred to as a model) to suit the problem to be solved is widely used, and a method of preparing multiple models and using them depending on the purpose, user level, etc. is known. Summary of the Invention [Problem to be solved by the invention]

[0003] If the feature values ​​of the inputs (explanatory variables) to the model are known, it is better to use a model trained using those feature values, but if the feature values ​​are unknown, it is better to use a model trained without those feature values.

[0004] For example, when creating a quote, consider a model that predicts the probability that the quote will be accepted. Customer information is an important input to the model in predicting the acceptance rate of the quote. Therefore, if there is quote data (training data) for the same customer (i.e., the same customer whose acceptance rate is being predicted) when training the model, the model trained with customer information will have higher prediction accuracy. However, when creating a quote for a new customer not included in the training data, the customer information of existing customers is likely to interfere with the prediction, and the prediction accuracy of the model trained with customer information will be lower (i.e., a model trained with only the quote content as a feature, rather than customer information, would be better).

[0005] Therefore, an object of the present invention is to improve the accuracy of prediction by using an appropriate model. [Means for solving the problem]

[0006] A service providing system according to one embodiment of the present invention is a service providing system including a service providing device and a user terminal, wherein the service providing device has a model memory unit that stores a first model trained using features of explanatory variables and a second model trained using invariant features, which are features of the explanatory variables to which no new values ​​are input during prediction, a learned feature value memory unit that stores trained feature values ​​of features other than the invariant features, and a model switching execution unit that performs prediction by using the first model when the trained feature values ​​are input and by using the second model when a value other than the trained feature values ​​is input, and outputs a predicted value, and the user terminal has a display means that displays the predicted value received from the service providing device. [Effects of the Invention]

[0007] According to the present invention, the accuracy of prediction can be improved by using an appropriate model. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining an overview of the present invention. [Figure 2] 1 is a diagram illustrating an overall configuration according to an embodiment of the present invention. [Figure 3] FIG. 2 is a hardware configuration diagram of a service providing device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a hardware configuration diagram of a user terminal according to an embodiment of the present invention. [Figure 5] 1 is a functional block diagram of a service providing system according to one embodiment of the present invention (first embodiment). [Figure 6] FIG. 1 is a sequence diagram showing a model switching process according to one embodiment of the present invention (first embodiment). [Figure 7] FIG. 10 is a functional block diagram of a service providing system according to one embodiment of the present invention (second embodiment). [Figure 8] FIG. 10 is a sequence diagram showing a model switching process according to one embodiment of the present invention (second embodiment). [Figure 9] 10 is an example of a screen displayed on a user terminal according to an embodiment of the present invention. [Figure 10] 10 is an example of a screen displayed on a user terminal according to an embodiment of the present invention. [Figure 11] 10 is an example of a screen displayed on a user terminal according to an embodiment of the present invention. [Figure 12] 10 is an example of a screen displayed on a user terminal according to an embodiment of the present invention. [Figure 13] FIG. 10 is a functional block diagram of a service providing system according to one embodiment of the present invention (third embodiment). [Figure 14] 10 is an example of data stored in a result storage unit according to an embodiment of the present invention. [Figure 15] FIG. 10 is a diagram for explaining calculation of model accuracy according to an embodiment of the present invention. [Figure 16] FIG. 10 is a sequence diagram showing a model switching process according to one embodiment of the present invention (third embodiment). DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] <Summary> 1 is a diagram illustrating an overview of the present invention. For example, a service providing device according to one embodiment of the present invention is a device (quote acceptance rate prediction device 11) that provides a service that predicts the acceptance rate of quotations. The quote acceptance rate prediction device 11 predicts the acceptance rate of quotations from quotations received from a user terminal 20 or information used when creating quotations, using trained models 1 and 2 generated by machine learning, and transmits the acceptance rate to the user terminal 20.

[0011] In this case, since the input patterns of materials and construction methods are predetermined, their influence on the order rate is considered to be invariant, and the estimate amount is also considered to be invariant. However, when the customer name is learned as one of the features during model training, if the same customer name (i.e., the same customer name as in the training data) is input, the prediction accuracy is high. However, if a new customer name that has not been learned is input, the customer name will become noise and the prediction accuracy will be low because it is not in the training data. Therefore, in one embodiment of the present invention, learned customer names are stored, and if the customer name input to the model during prediction is a learned customer name, Model 1, which was trained using the customer name as one of the features, is adopted to predict the order rate of the quote. Otherwise, Model 2, which was trained without using the customer name as a feature, is adopted.

[0012] <Overall structure> 2 is a diagram showing an overall configuration according to one embodiment of the present invention. The service providing system 1 can include a service providing device 10 and a user terminal 20. The service providing device 10 and the user terminal 20 are connected to a communication network.

[0013] <<Service providing device>> The service providing device (e.g., a server) 10 is a device that provides various services (e.g., a service for predicting the acceptance rate of quotes) using a trained model (hereinafter also referred to as a model) generated by machine learning. The service providing device 10 is one or more computers.

[0014] <<User terminal>> The user terminal 20 is a terminal (for example, a personal computer, a smartphone, or the like) operated by a person who receives a service (for example, a service for predicting an acceptance rate of a quote) provided by the service providing device 10.

[0015] Specifically, the service providing system 1 performs prediction and outputs a predicted value by switching between a first model trained using all feature quantities of the explanatory variables and a second model trained using only invariant feature quantities of the explanatory variables (note that invariant feature quantities are feature quantities among the feature quantities of the explanatory variables to which new values ​​are not input during prediction). The service providing system 1 uses the first model when trained feature quantities of feature quantities other than the invariant feature quantities are input, and uses the second model when a value other than the trained feature quantities is input.

[0016] The service providing device 10 may be constructed in a cloud environment, or may be constructed in the same on-premise environment as the user terminal 20. As in the second and third embodiments described later, the model may be in the service providing device 10, and the user terminal 20 may have a function for switching between the learned feature values ​​and the model.

[0017] The devices described in the example are merely one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, a service provider (e.g., a server) 10 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0018] <Hardware configuration> 3 is a hardware configuration diagram of a service providing device (for example, a server) 10 according to one embodiment of the present invention. The same applies to a user terminal (in the case of a personal computer).

[0019] As shown in FIG. 3, the service providing device 10 is constructed by a computer, and as shown in FIG. 3, it is equipped with a CPU 1001, a ROM 1002, a RAM 1003, a HD 1004, an HDD (Hard Disk Drive) controller 1005, a display 1006 (display means), an external device connection I / F (Interface) 1007, a network I / F 1008, a data bus 1009, a keyboard 1010, a pointing device 1011, a DVD-RW (Digital Versatile Disk Rewritable) drive 1013, and a media I / F 1015.

[0020] Of these, the CPU 1001 controls the overall operation of the service providing device 10. The ROM 1002 stores programs, such as an IPL, used to drive the CPU 1001. The RAM 1003 is used as a work area for the CPU 1001. The HD 1004 stores various data, such as programs. The HDD controller 1005 controls the reading and writing of various data from and to the HD 1004 under the control of the CPU 1001. The display 1006 (display means) displays various information, such as a cursor, menu, window, text, or image. The external device connection I / F 1007 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories, printers, etc. The network I / F 1008 is an interface for data communication using a communication network. The bus line 1009 is an address bus, data bus, etc. for electrically connecting the components, such as the CPU 1001, shown in FIG. 3.

[0021] The keyboard 1010 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 1011 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 1013 controls reading and writing of various data from a DVD-RW 1012, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 1015 controls reading and writing (storing) of data from a recording medium 1014, such as a flash memory.

[0022] FIG. 4 is a diagram showing the hardware configuration of a user terminal (in the case of a smartphone) 20 according to an embodiment of the present invention.

[0023] As shown in FIG. 4, the user terminal 20 includes a CPU 2001, a ROM 2002, a RAM 2003, an EEPROM 2004, a CMOS sensor 2005, an image sensor I / F 2006, an acceleration / direction sensor 2007, a media I / F 2009, and a GPS receiving unit 2011.

[0024] Of these, the CPU 2001 controls the overall operation of the user terminal 20. The ROM 2002 stores programs used to drive the CPU 2001, such as the CPU 2001 and IPL. The RAM 2003 is used as a work area for the CPU 2001. The EEPROM 2004 reads and writes various data, such as smartphone programs, under the control of the CPU 2001. The CMOS (Complementary Metal Oxide Semiconductor) sensor 2005 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) under the control of the CPU 2001 to obtain image data. Note that instead of a CMOS sensor, an imaging means such as a CCD (Charge Coupled Device) sensor may also be used. The imaging element I / F 2006 is a circuit that controls the operation of the CMOS sensor 2005. The acceleration / azimuth sensor 2007 is a type of sensor, such as an electronic magnetic compass or gyrocompass that detects geomagnetism, or an acceleration sensor. A media I / F 2009 controls reading and writing (storing) of data from and to a recording medium 2008 such as a flash memory. A GPS receiving unit 2011 receives GPS signals from GPS satellites.

[0025] The user terminal 20 also includes a long-distance communication circuit 2012, a CMOS sensor 2013, an image sensor I / F 2014, a microphone 2015, a speaker 2016, an audio input / output I / F 2017, a display 2018 (display means), an external device connection I / F (Interface) 2019, a short-distance communication circuit 2020, an antenna 2020a of the short-distance communication circuit 2020, and a touch panel 2021.

[0026] Among these, the long-distance communication circuit 2012 is a circuit that communicates with other devices via a communication network. The CMOS sensor 2013 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 2001. The imaging element I / F 2014 is a circuit that controls the operation of the CMOS sensor 2013. The microphone 2015 is a built-in circuit that converts sound into an electrical signal. The speaker 2016 is a built-in circuit that converts the electrical signal into physical vibrations to generate sounds such as music and voice. The audio input / output I / F 2017 is a circuit that processes the input and output of audio signals between the microphone 2015 and the speaker 2016 under the control of the CPU 2001. The display 2018 (display means) is a type of display means such as an LCD or organic EL (Electro Luminescence) display that displays images of subjects, various icons, etc. The external device connection I / F 2019 is an interface for connecting various external devices. The short-range communication circuit 2020 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 2021 is a type of input means that allows a user to operate the user terminal 20 by pressing the display 2018 (display means).

[0027] The user terminal 20 also includes a bus line 2010. The bus line 2010 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 2001 shown in FIG.

[0028] [Embodiment 1] The first embodiment will be described.

[0029] <Functional configuration> 5 is a functional block diagram of a service providing system 1 according to one embodiment of the present invention (embodiment 1). For example, the user terminal 20 creates an estimate and transmits the estimate to the service providing device 10, and the service providing device 10 predicts the acceptance rate of the estimate and transmits the acceptance rate of the estimate to the user terminal 20.

[0030] The service providing device 10 can include a model storage unit 101, a learned feature value storage unit 102, a prediction process execution unit 103, a feature calculation unit 104, and a model switching execution unit 105. The service providing device 10 can function as the prediction process execution unit 103, the feature calculation unit 104, and the model switching execution unit 105 by executing a program.

[0031] The model storage unit 101 stores a first model 111 trained using all feature quantities of explanatory variables, and a second model 112 trained using only invariant feature quantities of explanatory variables.

[0032] [Invariant features] Here, invariant features will be described. Invariant features are features of inputs (explanatory variables) to a model, to which new values ​​(i.e., values ​​not present in the learning data) are not input at the time of prediction. In other words, features other than invariant features are features to which new values ​​(i.e., values ​​not present in the learning data) are input at the time of prediction. For example, in this embodiment, the customer name in the estimate is assumed as a feature other than invariant features (a feature to which new values ​​(i.e., values ​​not present in the learning data) are input at the time of prediction). "Features other than invariant features" are determined in advance, and a model (second model) trained using only invariant features (for example, features other than the customer name in the estimate (in this case, the feature other than the invariant features is the customer name)) is generated.

[0033] Here are some examples of feature values ​​that do not become invariant features. Feature values ​​such as order quantity and price are possible, but these can usually be treated as invariant features because the meaning of the tendency of the value does not change whether it is large or small. However, if only biased training data is available, and the number of orders during model training ranged from 1 to 10, but there is concern that large orders of 10,000 may occur during prediction, then the number of orders will not be treated as an invariant feature. If an order number outside the learning range of the model's training data is received as a feature value, accuracy may be improved by using a training model that does not use order quantity as a feature.

[0034] The learned feature value storage unit 102 stores learned feature values ​​of features other than the invariant feature values ​​(that is, feature values ​​used when generating the first model).

[0035] The learned feature value storage unit 102 may store one or more values ​​of one or more types of feature. For example, assume that "Company A," "Company B," and "Company C" are stored as learned feature values ​​for customer names, and "Yamada Taro" and "Ricoh Taro" are stored as learned feature values ​​for staff names. In this case, when customer name: Company D and staff name: Yamada Hanako are input at the time of prediction, the second model is used.

[0036] The prediction processing execution unit 103 receives an input value from the user terminal 20, causes the feature calculation unit 104 to calculate the feature of the input value, causes the model switching execution unit 105 to make a prediction from the feature using the first model or the second model, and transmits the prediction result to the user terminal 20.

[0037] The feature calculation unit 104 calculates feature values ​​from the input values ​​received by the prediction process execution unit 103 from the user terminal 20 (converts the input values ​​into feature values). For example, the input values ​​may be used as they are as feature values, or the input values ​​of character strings may be converted into arbitrary numbers and used as feature values, or the number of uses, average, median, maximum value, minimum value, etc. may be calculated from the input values ​​of numbers and used as feature values.

[0038] For example, if the input value is Contact person: Taro Yamada ·Customer name: Company A Estimated price: 100,000 yen ·Materials used: A material / B material / C material ·Construction method used: A method / B method Delivery time: Yes When this is the case, the feature amount may be as follows: Contact person: Taro Yamada ·Customer name: Company A Estimated price: 100,000 ·Number of materials used: 3 ·Number of construction methods used: 2 Delivery time: 1

[0039] The model switching execution unit 105 uses the first model when a learned feature value of a feature other than an invariant feature is input, and uses the second model when a value other than the learned feature value is input. The model switching execution unit 105 inputs the feature value to the first model or the second model and causes it to output a predicted value (e.g., the acceptance rate of a quote).

[0040] <Method> FIG. 6 is a sequence diagram showing a model switching process according to one embodiment of the present invention (first embodiment).

[0041] In step 101 (S101), the user 21 inputs an instruction to the user terminal 20 to execute a service (for example, a service for predicting the acceptance rate of a quotation) using a trained model generated by machine learning.

[0042] In step 102 (S102), the user terminal 20 transmits an instruction to the prediction process execution unit 103 of the service providing device 10 to request execution of a service (e.g., a service for predicting the acceptance rate of a quotation) using the trained model generated by machine learning input in S101. Specifically, the user terminal 20 transmits an input value (e.g., quotation data) to the prediction process execution unit 103. Note that the data format of the input value may be any format.

[0043] In step 103 (S103), the prediction process execution unit 103 instructs the feature calculation unit 104 to calculate the feature of the input value received in S102.

[0044] In step 104 (S104), the feature calculation unit 104 calculates the feature of the input value (that is, the feature of the explanatory variable).

[0045] In step 105 (S105), the prediction process execution unit 103 instructs the model switching execution unit 105 to switch to a model according to the feature value calculated in S104 and execute prediction.

[0046] In step 106 (S106) and step 107 (S107), the model switching execution unit 105 refers to the learned feature amount value storage unit 102.

[0047] In step 108 (S108), the model switching execution unit 105 determines whether or not the feature values ​​of S105 include values ​​other than the learned feature values. If the feature values ​​of the input values ​​include values ​​other than the learned feature values, the process proceeds to step 109. If the feature values ​​of the input values ​​include values ​​other than the learned feature values, the process proceeds to step 110.

[0048] In step 109 (S109) and step 111 (S111), the model switching execution unit 105 acquires the first model stored in the model storage unit 101 (that is, the model trained using all the feature amounts of the explanatory variables).

[0049] In step 110 (S110) and step 111 (S111), the model switching execution unit 105 acquires the second model stored in the model storage unit 101 (i.e., a model trained using only the invariant features of the explanatory variables).

[0050] In step 112 (S112), the model switching execution unit 105 executes prediction using the model acquired in S111. Specifically, the model switching execution unit 105 inputs feature values ​​into the first model or the second model, and causes a predicted value (for example, the acceptance rate of the estimate) to be output.

[0051] In step 113 (S113), the model switching execution unit 105 transmits the predicted value of S112 (for example, the order acceptance rate of the estimate) to the prediction process execution unit 103.

[0052] In step 114 (S114), the prediction process executing unit 103 transmits the predicted value (for example, the acceptance rate of the quotation) of S113 to the user terminal 20.

[0053] In step 115 (S115), the user terminal 20 displays the predicted value (for example, the acceptance rate of the quotation) received in S114 on the display 1006 (display means) or the display 2018 (display means).

[0054] [Embodiment 2] A second embodiment will be described.

[0055] <Functional configuration> 7 is a functional block diagram of a service providing system 1 according to one embodiment of the present invention (embodiment 2). The following mainly describes the differences from embodiment 1. In embodiment 2, the user terminal 20 includes a learned feature value storage unit and determines whether to switch models.

[0056] The service providing device 10 can include a model storage unit 101, a prediction process execution unit 103, and a feature calculation unit 104. The service providing device 10 can function as the prediction process execution unit 103 and the feature calculation unit 104 by executing a program.

[0057] The user terminal 20 can include a learned feature value storage unit 102, a model switching execution unit 105, and a model switching condition input unit 106. The user terminal 20 can function as the model switching execution unit 105 and the model switching condition input unit 106 by executing a program.

[0058] [Service providing device] The model storage unit 101 is the same as in the first embodiment.

[0059] The prediction processing execution unit 103 receives an input value and a model specification from the user terminal 20, causes the feature calculation unit 104 to calculate the feature of the input value, makes a prediction from the feature using the first model or the second model, and transmits the prediction result to the user terminal 20.

[0060] The feature amount calculation unit 104 is the same as in the first embodiment.

[0061] [User device] The learned feature value storage unit 102 is the same as that in embodiment 1. Note that the learned feature value storage unit 102 may be provided in a device other than the user terminal 20 (for example, a device that provides an arbitrary service).

[0062] The learned feature value stored in the learned feature value storage unit 102 may be changed in response to an instruction input to the user terminal 20.

[0063] The model switching condition input unit 106 receives the switching conditions between the first model and the second model input by the user 21 to the user terminal 20 and stores them in an arbitrary storage unit. The switching conditions between the first model and the second model are changed in response to instructions input to the user terminal 20. In addition, the switching conditions between the first model and the second model are changed at predetermined intervals in response to instructions input to the user terminal 20.

[0064] [Model switching conditions] Here, the model switching condition will be described. The model switching condition is an arbitrary condition used for selecting a model. For example, the model switching condition is "select a model (second model) trained using only invariant features, even if the feature values ​​of the input values ​​are trained feature values," and the user 21 can specify one or more feature values. For example, the model switching condition is "select a first model at a predetermined time (e.g., time A), and select a second model at a predetermined time (e.g., time B)," and the user 21 can specify the predetermined time and the model to be selected at that time. Note that the model switching condition may always select one of the models (i.e., trained feature values ​​are not used).

[0065] The model switching execution unit 105 selects a model based on the switching conditions between the first model and the second model input by the user 21 to the user terminal 20, the learned feature values ​​(if necessary) stored in the learned feature value storage unit 102, and the input values. The model switching execution unit 105 transmits the input values ​​and the model designation (i.e., the selected model) to the service providing device 10, and receives the prediction results from the service providing device 10. By changing the switching conditions in this way, it is possible to check the prediction results for multiple models without re-learning or additional learning of the models.

[0066] For example, if the model switching condition is "select a model (second model) trained using only invariant features even if the feature value of the input value is a learned feature value," and the feature values ​​are company A and company C, when the input value is company A, the model switching execution unit 105 selects a model (second model) trained using only invariant features without referring to the learned feature value storage unit 102. When the input value is company B, the model switching execution unit 105 refers to the learned feature value storage unit 102, and if company B is a learned feature value, selects a model (first model) trained using all features, or if company B is not a learned feature value, selects a model (second model) trained using only invariant features.

[0067] <Method> FIG. 8 is a sequence diagram showing a model switching process according to one embodiment of the present invention (second embodiment).

[0068] In step 201 (S201), the user 21 inputs a switching condition between the first model and the second model into the user terminal 20. S201 may be executed when the system starts operation, when a change in the switching condition is required, or for each prediction.

[0069] The user 21 who sets the switching conditions is not limited to the person who prepares the estimate, but may be a system administrator, designer, or the like.

[0070] In step 202 (S202), the user 21 inputs an instruction to the user terminal 20 to execute a service (e.g., a service for predicting the acceptance rate of a quotation) using the trained model generated by machine learning. Specifically, the user 21 inputs an input value (e.g., quotation data) into the user terminal 20.

[0071] In step 203 (S203) and step 204 (S204), the model switching execution unit 105 acquires the switching conditions between the first model and the second model (that is, the switching conditions input by the model switching condition input unit 106 in S201).

[0072] In step 205 (S205) and step 206 (S206), if the learned feature amount values ​​are necessary, the model switching execution unit 105 acquires the learned feature amount values ​​stored in the learned feature amount value storage unit 102.

[0073] In step 207 (S207), the model switching execution unit 105 selects a model based on the switching condition in S201, the learned feature values ​​in S205 and S206 (if necessary), and the input value in S202.

[0074] In step 208 (S208), the model switching execution unit 105 transmits the input value of S202 and the model designation (that is, the model selected in S207) to the service providing device 10.

[0075] In step 209 (S209), the prediction process execution unit 103 instructs the feature amount calculation unit 104 to calculate the feature amount of the input value received in S208.

[0076] In step 210 (S210), the feature calculation unit 104 calculates the feature of the input value (that is, the feature of the explanatory variable).

[0077] In step 211 (S211) and step 212 (S212), the prediction process executing unit 103 acquires the model designated by the user terminal 20 from the first model and the second model stored in the model storage unit 101.

[0078] In step 213 (S213), the prediction process execution unit 103 executes prediction using the model acquired in S212. Specifically, the prediction process execution unit 103 inputs feature values ​​into the first model or the second model, and causes the model to output a predicted value (for example, the acceptance rate of the estimate).

[0079] In step 214 (S214), the prediction process executing unit 103 transmits the predicted value (for example, the acceptance rate of the quotation) of S213 to the user terminal 20.

[0080] In step 215 (S215), the user terminal 20 displays the predicted value (for example, the acceptance rate of the quotation) received in S214 on the display 1006 (display means) or the display 2018 (display means).

[0081] <User Interface> Fig. 9 is an example of a screen displayed on the user terminal 20 according to one embodiment of the present invention. Fig. 9 is a screen for the user 21 to input into the user terminal 20 an instruction to execute a service for predicting the acceptance rate of a quotation. As shown in Fig. 9, quotation information is displayed, and when the "Acceptance Rate Prediction" button is pressed, the acceptance rate is predicted and displayed.

[0082] Fig. 10 is an example of a screen displayed on the user terminal 20 according to an embodiment of the present invention. Fig. 10 is a screen showing the predicted acceptance rate of quotes.

[0083] Fig. 11 is an example of a screen displayed on the user terminal 20 according to one embodiment of the present invention. Fig. 11 is a screen for the user 21 to input model switching conditions. Note that in Fig. 11, when the model switching execution unit 105 specifies a model to the prediction processing execution unit 103, the switching is performed by specifying the model version rather than information on whether all features or only invariant features are to be used.

[0084] Assume that "customer name" is a feature for which an unknown value may be input. As shown in FIG. 11, learned customer names are displayed. The user 21 can delete unnecessary customer names from the learned customer names. This allows the feature values ​​with a small population at the time of learning and close to noise to be excluded.

[0085] If the trained customer name is Company A, when predicting the acceptance rate of a quotation, model 1.0, which is the "customer trained AI version," is used if the customer name is Company A, and model 1.1, which is the "default AI version," is used if the customer name is other than Company A.

[0086] By unchecking the "Switch AI version according to customer" checkbox, you have entered a switching condition that always selects the "default AI version" without using the learned feature values.

[0087] Fig. 12 is an example of a screen displayed on the user terminal 20 according to one embodiment of the present invention. Fig. 12 is a screen for the user 21 to input model switching conditions (specifically, switching conditions according to time). Note that in Fig. 12, when the model switching execution unit 105 specifies a model to the prediction processing execution unit 103, the switching is performed by specifying the model version rather than information on whether all features or only invariant features are to be used.

[0088] The user 21 can specify the version of the model to be used for prediction at a predetermined time (for example, each month). For example, a version of the model trained using all feature quantities and a version of the model trained using only invariant feature quantities are specified at a predetermined time (for example, each month). This makes it possible to deal with situations in which the accuracy and tendency of model predictions change due to seasonal factors such as busy periods.

[0089] In this way, in the second embodiment, the user 21 can specify the model switching conditions. For example, in the case of customer names, relationships and tendencies differ for each customer, so for customers with learned customer names, a model trained using all feature quantities can be expected to improve prediction accuracy. However, the relationship with a customer may change due to social conditions or factors over time. When the relationship with a customer changes in this way, the prediction accuracy of a model trained using all feature quantities (i.e., a model that has already learned customer names) may decrease. Therefore, when the relationship with a customer temporarily changes, a model trained using all feature quantities is inappropriate, and a decrease in prediction accuracy can be prevented by selecting a model trained using only invariant feature quantities.

[0090] It is possible to maintain prediction accuracy by relearning the model using data from a certain period after the system has been put into operation, or by additionally training specific features. However, relearning or additional training requires preparing a new configuration, which takes time and is currently difficult for a typical user to perform. Furthermore, if the cause of the decrease in prediction accuracy is more likely to be temporary noise, relearning or additional training may be inappropriate. In the second embodiment, it is possible to prevent a decrease in prediction accuracy without relearning or additional training.

[0091] [Embodiment 3] A third embodiment will now be described.

[0092] <Functional configuration> 13 is a functional block diagram of a service providing system 1 according to one embodiment of the present invention (Embodiment 3). The following mainly describes the differences from Embodiment 2. In Embodiment 3, the user terminal 20 switches models based on the accuracy of the models.

[0093] The service providing device 10 can include a model storage unit 101, a prediction process execution unit 103, and a feature calculation unit 104. The service providing device 10 can function as the prediction process execution unit 103 and the feature calculation unit 104 by executing a program.

[0094] The user terminal 20 can include a learned feature value storage unit 102, a model switching execution unit 105, a model switching condition input unit 106, a model accuracy calculation unit 107, and a result storage unit 108. The user terminal 20 can function as the model switching execution unit 105, the model switching condition input unit 106, and the model accuracy calculation unit 107 by executing a program.

[0095] [Service providing device] The model storage unit 101 is the same as in the first embodiment.

[0096] The prediction processing execution unit 103 receives an input value and a model specification from the user terminal 20, causes the feature calculation unit 104 to calculate the feature of the input value, makes a prediction from the feature using the first model or the second model, and transmits the prediction result to the user terminal 20.

[0097] The feature amount calculation unit 104 is the same as in the first embodiment.

[0098] [User device]

[0099] The model switching execution unit 105 switches the model based on the prediction accuracy of the first model calculated by the model accuracy calculation unit 107. Specifically, if the prediction accuracy of the first model (model trained using all feature quantities) is below a threshold, the model switching execution unit 105 selects the second model (model trained using only invariant feature quantities), and if the prediction accuracy of the first model (model trained using all feature quantities) is equal to or greater than the threshold, the model switching execution unit 105 selects a model in accordance with the model switching conditions accepted by the model switching condition input unit 106, as in the second embodiment. The model switching execution unit 105 transmits the input values ​​and the model specification (i.e., the selected model) to the service providing device 10 and receives the prediction result from the service providing device 10.

[0100] The model switching execution unit 105 may determine whether to switch models based only on the prediction accuracy of the first model, without using a switching condition.

[0101] The result storage unit 108 stores the prediction result of the first model, the prediction target information, and the evaluation result.

[0102] The model accuracy calculation unit 107 calculates the accuracy of the prediction of the first model based on the prediction result of the first model, the prediction target information, and the evaluation result.

[0103] The model switching condition input unit 106 is the same as in the first embodiment.

[0104] The learned feature value storage unit 102 is the same as that in embodiment 1. Note that the learned feature value storage unit 102 may be provided in a device other than the user terminal 20 (for example, a device that provides an arbitrary service).

[0105] 14 shows an example of data stored in the result storage unit 108 according to one embodiment of the present invention. The result storage unit 108 stores the prediction results of the first model, prediction target information, and evaluation results for an arbitrary period (e.g., the most recent period). The data in the result storage unit 108 may be stored each time a prediction is made, or may be stored at a predetermined timing.

[0106] The predicted result is the result predicted by the first model (eg, the order acceptance rate (eg, a percentage)).

[0107] The prediction target information is information for identifying input values ​​(for example, ID, person in charge name, customer name, estimated amount, materials used, construction method used, delivery date). The prediction target information may be the same as the data used for prediction, or other information may be added to the data used for prediction. Only the ID may be managed.

[0108] The evaluation result indicates whether the result was as predicted (for example, won or lost) ("Status" in FIG. 14). Note that the result is not limited to character strings such as won or lost, but may also be a boolean value or a numeric value.

[0109] 15 is a diagram for explaining calculation of model accuracy according to one embodiment of the present invention. A model accuracy calculation unit 107 calculates the accuracy of prediction of the first model based on the prediction result of the first model corresponding to the input value at the time of prediction and the evaluation result.

[0110] For example, assume that the learned feature value storage unit 102 stores the learned feature value "Company A" for the feature "Customer Name." If the input value at the time of prediction is Company A, the model accuracy calculation unit 107 acquires the prediction result and evaluation result for which the customer name is Company A from the result storage unit 108. For example, the model accuracy calculation unit 107 processes the prediction result into a format suitable for calculating the accuracy of the model. For example, if the "prediction result" is an order rate, the prediction result is processed into the format "Order received" if the order rate is equal to or greater than a threshold, and "Lost" if the order rate is less than the threshold ("Predicted Order Result" in FIG. 15). Next, the model accuracy calculation unit 107 compares the processed prediction result ("Predicted Order Result" in FIG. 15) suitable for calculating the accuracy of the model with the evaluation result ("Status" in FIG. 15) to make a judgment (success or failure of "Judgment" in FIG. 15). For example, the accuracy of the model is calculated as "Model accuracy = Number of successful judgments / Number of judgment targets."

[0111] <Method> FIG. 16 is a sequence diagram showing a model switching process according to one embodiment of the present invention (third embodiment).

[0112] In step 301 (S301), the user 21 inputs a switching condition between the first model and the second model into the user terminal 20. S301 may be executed when the system starts operation, when a change in the switching condition is required, or for each prediction.

[0113] The user 21 who sets the switching conditions is not limited to the person who prepares the estimate, but may be a system administrator, designer, or the like.

[0114] In step 302 (S302), the user 21 inputs an instruction to the user terminal 20 to execute a service (e.g., a service for predicting the acceptance rate of a quotation) using the trained model generated by machine learning. Specifically, the user 21 inputs an input value (e.g., quotation data) into the user terminal 20.

[0115] In step 303 (S303), the model switching execution unit 105 instructs the model accuracy calculation unit 107 to calculate the accuracy of the first model corresponding to the input value in S302.

[0116] In step 304 (S304) and step 305 (S305), the model accuracy calculation unit 107 acquires, from the result storage unit 108, the prediction results and evaluation results corresponding to the input values ​​in S302.

[0117] In step 306 (S306), the model accuracy calculation unit 107 calculates the accuracy of the first model corresponding to the input value in S302.

[0118] In step 307 (S307), the model accuracy calculation unit 107 transmits the accuracy of the first model calculated in S306 to the model switching execution unit 105. If the accuracy of the first model is less than the threshold, the process proceeds to step 308;

[0119] In step 308 (S308), the model switching execution unit 105 transmits the input value of S302 and the model specification (the second model (the model trained using only the invariant features)) to the service providing device 10.

[0120] In step 309 (S309) and step 310 (S310), the model switching execution unit 105 acquires the switching conditions between the first model and the second model (that is, the switching conditions input by the model switching condition input unit 106 in S301).

[0121] In step 311 (S311) and step 312 (S312), if the learned feature amount values ​​are necessary, the model switching execution unit 105 acquires the learned feature amount values ​​stored in the learned feature amount value storage unit 102.

[0122] In step 313 (S313), the model switching execution unit 105 selects a model based on the switching condition in S301, the learned feature values ​​in S311 and S312 (if necessary), and the input value in S302.

[0123] In step 314 (S314), the model switching execution unit 105 transmits the input value in S302 and the model designation (that is, the model selected in S313) to the service providing device 10.

[0124] In step 315 (S315), the prediction process execution unit 103 instructs the feature amount calculation unit 104 to calculate the feature amount of the input value received in S314.

[0125] In step 316 (S316), the feature calculation unit 104 calculates the feature of the input value (that is, the feature of the explanatory variable).

[0126] In step 317 (S317) and step 318 (S318), the prediction process executing unit 103 acquires the model specified by the user terminal 20 from the first model and the second model stored in the model storage unit 101.

[0127] In step 319 (S319), the prediction process execution unit 103 executes prediction using the model acquired in S318. Specifically, the prediction process execution unit 103 inputs feature values ​​into the first model or the second model, and causes the model to output a predicted value (for example, the acceptance rate of the estimate).

[0128] In step 320 (S320), the prediction process executing unit 103 transmits the predicted value (for example, the acceptance rate of the quotation) of S319 to the user terminal 20.

[0129] In step 321 (S321), the user terminal 20 displays the predicted value (for example, the acceptance rate of the quotation) received in S320 on the display 1006 (display means) or the display 2018 (display means).

[0130] In step 322 (S322), the prediction result, the prediction target information, and the evaluation result are stored in the result storage unit .

[0131] <Effects> In this way, in one embodiment of the present invention, when predicting the acceptance rate of quotations from existing customers (customers that have already been trained), a model trained using all of the features of the explanatory variables is adopted, and when predicting the acceptance rate of quotations from new customers (customers that have not already been trained), a model trained using only the invariant features of the explanatory variables is adopted, thereby improving the accuracy of predictions.

[0132] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above. [Explanation of symbols]

[0133] 1. Service provision system 10 Service providing device 11. Estimate / Order Rate Prediction Device 20 User terminal 21 users 101 Model memory section 111 First model (all features) 112 Second model (invariant features only) 102 Learned feature value memory unit 121 Learned feature values 103 Prediction Processing Execution Unit 104 Feature Calculation Unit 105 Model switching execution unit 106 Model switching condition input section 107 Model Accuracy Calculation Unit 108 Result storage section 181 Prediction Results 182 Forecast target information 183 Evaluation Results [Prior art documents] [Patent documents]

[0134] [Patent Document 1] Patent No. 6670916

Claims

1. A service providing system including a service providing device and a user terminal, the service providing device, a model storage unit that stores a first model trained using feature quantities of explanatory variables and a second model trained using invariant feature quantities, which are feature quantities of the explanatory variables to which no new values ​​are input at the time of prediction; a learned feature value storage unit in which learned feature values ​​of features other than the invariant feature are stored; a model switching execution unit that executes prediction by using the first model when the learned feature value is input, and by using the second model when a value other than the learned feature value is input, and outputs a predicted value; and The user terminal a display means for displaying the predicted value received from the service providing device; A service provision system having the above.

2. The service providing system according to claim 1 , wherein the learned feature value stored in the learned feature value storage unit is changed in response to an instruction input to the user terminal.

3. a model switching condition input unit that receives an input of a switching condition between the first model and the second model; The service providing system according to claim 1 , wherein the model switching execution unit selects a model based on the switching condition and an input value.

4. The service providing system according to claim 3 , wherein the condition for switching between the first model and the second model is changed in response to an instruction input to the user terminal.

5. 4. The service providing system according to claim 3, wherein the condition for switching between the first model and the second model is changed at predetermined intervals in response to an instruction input to the user terminal.

6. a result storage unit in which the prediction result of the first model, prediction target information, and evaluation result are stored; a model accuracy calculation unit that calculates the accuracy of prediction of the first model based on the prediction result, the prediction target information, and the evaluation result; The service providing system according to claim 1 , wherein the model switching execution unit switches the model based on the accuracy of the prediction.

7. The feature other than the invariant feature is the customer name of the estimate, The service providing system according to claim 1 , wherein the first model and the second model are models for predicting an acceptance rate of the quote.

8. A method executed by a service providing system including a service providing device and a user terminal, the service providing device, a model storage unit that stores a first model trained using feature quantities of explanatory variables and a second model trained using invariant feature quantities, which are feature quantities of the explanatory variables to which no new values ​​are input at the time of prediction; a learned feature value storage unit in which learned feature values ​​of features other than the invariant feature are stored, the service providing device, performing prediction by using the first model when the learned feature value is input, and by using the second model when a value other than the learned feature value is input, and outputting a predicted value; The user terminal A method for displaying the predicted value received from the service provider.

9. a model storage unit that stores a first model trained using feature quantities of explanatory variables and a second model trained using invariant feature quantities, which are feature quantities of the explanatory variables to which no new values ​​are input at the time of prediction; a learned feature value storage unit that stores learned feature values ​​of features other than the invariant feature, executing a process of performing prediction by using the first model when the learned feature value is input, and by using the second model when a value other than the learned feature value is input, and outputting a predicted value; On the user terminal, a program for executing a process of displaying the predicted value received from the service providing device;

10. a model storage unit that stores a first model trained using feature quantities of explanatory variables and a second model trained using invariant feature quantities, which are feature quantities of the explanatory variables to which no new values ​​are input at the time of prediction; a learned feature value storage unit in which learned feature values ​​of features other than the invariant feature are stored; a model switching execution unit that executes prediction by using the first model when the learned feature value is input, and by using the second model when a value other than the learned feature value is input, and outputs a predicted value; A service providing device comprising:

11. A method executed by a service providing device, the service providing device, a model storage unit that stores a first model trained using feature quantities of explanatory variables and a second model trained using invariant feature quantities, which are feature quantities of the explanatory variables to which no new values ​​are input at the time of prediction; a learned feature value storage unit in which learned feature values ​​of features other than the invariant feature are stored, A method for performing prediction by using the first model when the learned feature value is input, and by using the second model when a value other than the learned feature value is input, and outputting a predicted value.

12. a model storage unit that stores a first model trained using feature quantities of explanatory variables and a second model trained using invariant feature quantities, which are feature quantities of the explanatory variables to which no new values ​​are input at the time of prediction; a learned feature value storage unit that stores learned feature values ​​of features other than the invariant feature, A program for executing a process of making a prediction by using the first model when the learned feature value is input, and by using the second model when a value other than the learned feature value is input, and outputting a predicted value.

Images