Service provision system, service provision method, and program

Abstract

To perform an appropriate model evaluation when a model such as generative AI is used.SOLUTION: A system includes: a model evaluation response correctness collection part 306 in which a plurality of registered questions are input for each of a plurality of prepared models, and for collecting correctness of responses to an output question; an item response theory calculation part 307 for calculating an evaluation value for the plurality of models by conducting a calculation process according to an item response theory on the correctness data collected by the model evaluation response correctness collection part 306; and model selection part 308 for selecting the model to apply when providing a service according to the evaluation value calculated by the item response theory calculation part 307.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a service providing system, a service providing method, and a program. [Background technology]

[0002] Currently, much research is being conducted into generative AI (Artificial Intelligence) technology, and chatbots that use AI to generate appropriate answer texts to answer user questions are emerging. Generative AI uses large amounts of training data to learn and create a model, which is then used to generate the desired data, but the quality of the AI ​​depends on the training data used for learning and other factors. For this reason, it is necessary to evaluate the quality of the models created through learning and adopt the model with the best quality.

[0003] Patent Document 1 describes a technology that stores multiple trained models in a database and selects and uses the optimal model based on the performance of each model calculated using test data. Patent Document 2 describes a technique for using item response theory to evaluate the ability of an examinee taking an academic ability test. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 142766 [Patent Document 2] Patent No. 3645901 Summary of the Invention [Problem to be solved by the invention]

[0005] Previous evaluations of AI models, such as the method described in Patent Document 1, have had the problem that the results for all test data are evaluated uniformly. For example, if there is a bias in the questions, such as many similar questions appearing, the importance of each test item in the evaluation may differ, and simply counting the number of correct answers may not provide an appropriate evaluation. Furthermore, depending on the AI ​​model, there may be some things that it is good at and some that it is not good at, making it difficult to simply evaluate which model is better based on the number of correct answers alone.

[0006] It should be noted that there is a technique called Item Response Theory (IRT) described in Patent Document 2 as a technique for statistically processing and evaluating the results of human tests and questionnaires. However, currently, generative AI models are evaluated by simply counting the number of correct answers, which often makes it impossible to properly evaluate the generative AI models. While we have discussed the issues involved in evaluating generative AI models here, similar problems arise when evaluating models when performing processing using various models, such as machine learning.

[0007] The present invention aims to provide a service providing system, a service providing method, and a program that enable appropriate model evaluation when using a model such as generative AI. [Means for solving the problem]

[0008] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above-mentioned problems, and one example is a service provision system that includes a model evaluation answer correctness collection unit that inputs multiple registered questions for each of multiple prepared models and collects the correctness of the answers to the output questions, an item response theory calculation unit that performs calculation processing based on item response theory on the correctness data collected by the model evaluation answer correctness collection unit to calculate evaluation values ​​for multiple models, and a model selection unit that selects a model to be applied when providing a service based on the evaluation values ​​calculated by the item response theory calculation unit. [Effects of the Invention]

[0009] According to the present invention, even if a model is applied to a service in which the importance of evaluation differs depending on the test item, an appropriate model can be selected. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a configuration diagram showing an example of a usage pattern of a service providing system according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of the hardware configuration of a computer that constitutes a service providing system according to a first embodiment of the present invention. [Figure 3] 1 is a configuration diagram showing an example of a processing unit and a database prepared in an auxiliary storage device of a computer as a service providing system according to a first embodiment of the present invention. [Figure 4] FIG. 3 is a diagram showing an example of a question according to the first embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing examples of correct answers and incorrect answers to questions for each model according to the first embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing an example of an evaluation value for each model according to the first embodiment of the present invention. [Figure 7] FIG. 3 is a diagram showing an example of a question evaluation value according to the first embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing an example of a service provision screen according to the first embodiment of the present invention. [Figure 9] 5 is a flowchart showing the flow of processing when a question is made in the service providing system according to the first embodiment of the present invention. [Figure 10] 5 is a flowchart showing the flow of processing by a model evaluation processing unit according to the first embodiment of the present invention. [Figure 11] 10 is a flowchart showing the flow of processing by a model evaluation answer correct / incorrect collection processor according to the first embodiment of the present invention. [Figure 12] 10 is a flowchart showing the flow of processing by an additional model evaluation processing unit according to a second embodiment of the present invention. [Figure 13] 10 is a flowchart showing the flow of processing by an additional model evaluation answer correct / incorrect collection processing unit according to the second embodiment of the present invention. [Figure 14] 10A and 10B are diagrams showing examples of screens displaying the evaluation values ​​of questions and the correctness of models as modified examples of the embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] <First embodiment> A service providing system, a service providing method, and a program according to a first embodiment of the present invention will be described below with reference to FIGS.

[0012] [System Configuration] This embodiment is a service provision system provided to residents by government offices such as ward offices and city halls, and the government offices are configured as digital counter systems that provide resident services and accept application forms to residents using information processing terminals. This digital service center system uses AI technology to answer questions entered by residents and provides appropriate services and application forms. This model is capable of providing appropriate answers when a question is entered. For example, various types of models are possible, such as an AI model when generative AI technology is used, or a machine learning model when analytical AI technology is used.

[0013] FIG. 1 shows an example of the configuration of a digital window system as a service providing system according to this embodiment. As shown in FIG. 1, the digital window system comprises a digital window server 101 , a network 102 , and one or more user terminals 103 . The digital window server 101 is a server-type information processing device that operates based on a request from a user terminal 103 . The network 102 connects the digital window server 101 and the user terminal 103, enabling data communication between the devices, and the Internet, an intranet, or the like can be used.

[0014] The user terminal 103 is an information processing terminal such as a personal computer (PC), a tablet terminal, or a smartphone, and is a device for using services using a browser, etc. The user terminal 103 is connected to the digital window server 101 via a network. The user terminal 103 may be any information processing terminal on which a general web browser operates, and there may be a plurality of terminals, for example, each user may use a different terminal.

[0015] The digital window server 101 includes a model evaluation unit 305, a model database 314, and model answer correct / incorrect data 316. The processing units included in the digital window server 101 will be described in detail later.

[0016] [Hardware configuration of the digital window server] 2 shows an example of the configuration of the digital window server 101. The digital window server 101 is configured with the computer shown in FIG. That is, the computer constituting the digital window server 101 has a CPU (Central Processing Unit) 201 , a main memory device 202 , a network I / F 203 , a display device 204 , an input device 205 , and an auxiliary memory device 206 . The CPU 201 executes a program stored in the main storage device 202 or the auxiliary storage device 206 on the main storage device 202, and executes processing functions according to the program.

[0017] The main storage device 202 is configured as, for example, a RAM (Random Access Memory), and stores programs and calculation result data, and also provides the CPU 201 with a work area necessary for each process. The network I / F 203 connects the digital window server 101 to the network 102 (FIG. 1) and executes data transfer processing. The display device 204 is used by the administrator of the digital window server 101, and displays the model evaluations described below. The input device 205 is an input device such as a keyboard or a mouse used by the administrator of the digital window server 101 or the like.

[0018] The auxiliary storage device 206 is configured by a hard disk drive (HDD) or a solid state drive (SSD), and stores programs as well as data required for calculations, calculation result data, and data as various databases. In this embodiment, the program stored in the auxiliary storage device 206 is read into the main storage device 202 and executed under the control of the CPU 201. In this way, the digital window server 101 is configured with various processing function units.

[0019] Note that configuring the computer serving as the digital window server 101 as a computer with the configuration shown in Figure 2 is just one example, and some or all of the processing may be configured using hardware such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0020] [Configuration of the digital window server in terms of functions] FIG. 3 shows a database provided in the storage area of ​​the auxiliary storage device 206 and a processing unit configured in the digital window server 101 that reads and executes the programs and data stored in the database. In Figure 3, the auxiliary storage device 206 is shown as having each processing unit, but in reality, the processing units of the digital window server 101 are formed by executing the programs stored in the auxiliary storage device 206 under the control of the CPU 201.

[0021] The auxiliary storage device 206 of the digital window server 101 includes a resident access processing unit 301, a question and answer processing unit 302, a service provision processing unit 303, an application reception processing unit 304, and a model evaluation unit 305. The auxiliary storage device 206 also stores a resident information database 311, multiple resident services 312, multiple application formats 313, a model database 314, a model evaluation question list 315, model answer correct / incorrect data 316, and model evaluation results 317.

[0022] The resident access processing unit 301 provides a function that enables residents to connect to the digital window server 101 from the user terminal 103. That is, the resident access processing unit 301 provides services such as providing information to residents, resident services, and application forms to residents using web pages, etc. When a resident accesses the system, the resident access processing unit 301 provides a login function using an ID, password, or My Number card.

[0023] The question and answer processing unit 302 generates an answer to the question entered by the resident using a model selected from the generation AI models stored in the model database 314, and transmits the generated answer to the user terminal 103 for display. Furthermore, the question and answer processing unit 302 calls the service provision processing unit 303 if necessary to provide resident services 312. The question and answer processing unit 302 also calls the application reception processing unit 304 to send an application form format 313 to the user terminal 103 for display. If a resident is logged in, the question and answer processing unit 302 obtains information about the resident asking the question from the resident information database 311, and inputs the question and the resident information together into the generation AI to obtain an answer. The question and answer processing unit 302 then transmits the obtained answer to the user terminal 103 to display it.

[0024] The service provision processing unit 303 executes processing to provide a resident service 312 in response to an instruction from the user terminal 103 . The application reception processing unit 304 provides the application format 313 to the user terminal 103 for display, and executes processing to receive the content entered into the application format 313 by the resident on the user terminal 103 .

[0025] The digital window server 101 described so far is a server that performs processing to provide resident services to residents (user terminals 103), but the digital window server 101 of this embodiment also includes a model evaluation unit 305. The model evaluation unit 305 evaluates each model stored in the model database 314 and executes a model evaluation process to select a model based on the evaluation results. The model evaluation unit 305 includes a model evaluation answer correct / incorrect collection unit 306 , an item response theory calculation unit 307 , and a model selection unit 308 .

[0026] The model evaluation answer correctness collection unit 306 gives each question included in the model evaluation question list 315 to each model stored in the model database 314, and stores the correctness of the answers to each question for each model in the model answer correctness data 316 (model evaluation answer correctness collection process).

[0027] For example, the accuracy of an answer to each question is determined as "correct" if it matches the correct answer stored in the model evaluation question list 315. Alternatively, the accuracy of an answer can be determined by using natural language processing, generation AI, or classification AI, such as comparing included words and determining similar ones as "correct." Furthermore, multiple determination processes can be applied, such as determining based on the results of visual confirmation by an administrator of the digital window server 101. Furthermore, these multiple determination processes may be applied in combination.

[0028] The item response theory calculation unit 307 applies item response theory (IRT) to the model answer correct / incorrect data 316 to calculate the evaluation value of each model, as well as the difficulty and discriminatory power of each question, and stores them in the model evaluation results 317 (item response theory calculation processing). There are already tools available for calculating item response theory, such as the statistical software "R" and libraries for Python, and the digital window server 101 may use these to configure the item response theory calculation unit 307.

[0029] The model selection unit 308 provides a function of selecting a model to be used in the question and answer processing unit 302 thereafter based on the model evaluation result 317 . As a method for selecting a model, for example, the model selection unit 308 selects the model with the highest evaluation value calculated by the item response theory calculation unit 307. Alternatively, the model selection unit 308 may use a method in which an administrator of the digital window server 101 or the like views the calculation results of the item response theory calculation unit 307 and makes a selection, or a method in which the administrator selects the result of selecting the model with the highest evaluation value and then modifies it.

[0030] [Model evaluation question list] FIG. 4 shows an example of a model evaluation questionnaire 315. The model evaluation question list 315 has fields for each question: question text 401, resident attributes 402, and correct answer 403. The question text 401 stores a question that is assumed to be entered by a resident, such as "I'm moving to Ward A."

[0031] Resident attributes 402 stores information obtained from the resident information database and input into the AI ​​model, such as "annual income of 4 million yen." The correct answer 403 stores the expected answer when the question text 401 and the resident attributes 402 are input to the model. As an example of an expected answer, "Please submit a moving-out notification" is stored as an answer to the question "I'm moving to Ward A."

[0032] [Model answer correct / incorrect data] FIG. 5 shows an example of the model answer correct / incorrect data 316. The model answer correctness data 316 stores the correctness or incorrectness of the answers given by each model 501 included in the model database 314 to each question 502 included in the model evaluation question list 315 .

[0033] For example, as model answer correctness data 316, question 1 of model 1 is stored as "○", question 2 of model 1 is stored as "○", ... question m of model 1 (m is an integer corresponding to the number of questions) is stored as "○". In other words, as model answer correctness data 316, the correctness of each of questions 1 to m is stored for model 1 to model n (n is an integer corresponding to the number of models). In FIG. 5, for ease of explanation, correct and incorrect answers are indicated by ○ and ×, but in actual data, it is preferable to store the answers as Boolean values ​​such as True and False, or values ​​such as 0 and 1.

[0034] [Model evaluation results] 6 and 7 show examples of the model evaluation results 317. The model evaluation result 317 consists of model evaluation value data 600 and question evaluation value data 700 . FIG. 6 shows an example of model evaluation value data 600. The model evaluation value data 600 is data that stores the evaluation value 601 for each model calculated using the item response theory calculation unit 307 .

[0035] 6, the model evaluation value data 600 stores the evaluation value of each model, such as 1.6 for model 1, -0.7 for model 2, 0.4 for model n, etc. In this example, the higher the value, the higher the evaluation value.

[0036] FIG. 7 shows an example of question evaluation value data 700. The question evaluation value data 700 is data that stores the question evaluation value 701 for each question calculated using the item response theory calculation unit 307 . The question evaluation value 701 is composed of a difficulty level 702 and a discrimination ability 703 . The difficulty level 702 is a value that evaluates the difficulty of the question calculated using the item response theory calculation unit 307 .

[0037] The discrimination power 703 is a value that evaluates the ability to discriminate between the evaluations of the question model calculated using the item response theory calculation unit 307 . As shown in FIG. 7, the difficulty level 702 and discrimination power 703 are stored for each question, such as question 1, question 2, . . . , question m.

[0038] [Digital window system screen] FIG. 8 shows an example of a screen 800 of the user terminal 103 using the digital window server 101. The screen 800 of the user terminal 103 is arranged with a question input area 801, a question button 802, a chat display area 803, a resident service selection area 804, an application form selection area 805, a link 806 to return to the menu, a logout link 807, and the like.

[0039] The question input area 801 is an area where residents can input questions to the system in text format. A question button 802 is a button that is pressed after a question has been entered in the question input area 801 to input the question to the selected model. The chat display area 803 is an area where the entered question and the AI's answer are displayed. In the example of Fig. 8, they are displayed in a speech bubble format.

[0040] For example, if a user enters "I'm moving to Ward A," the AI's response to that question will be "Please submit a moving-out notification" and "Click here for the input format for the moving-out notification." The answer "Click here for the input format for the moving-out notification" is underlined to indicate that it is a link.

[0041] The resident service selection area 804 is provided with the expectation that residents will select services themselves. That is, the resident service selection area 804 displays a list of links to selectable resident services, and when a resident clicks on the link to the resident service they want to select, the service provision processing unit 303 is called and the corresponding resident service 312 is provided. For example, a list of resident services with links is displayed, such as "Moved," "Married," "Having a child," etc.

[0042] The application form selection area 805 is provided on the assumption that residents will select an application form themselves. That is, the application form selection area 805 displays a list of links to selectable application forms, and when a resident clicks on the link to the application form they want to select, the application reception processing unit 304 is called and the corresponding application form format 313 is displayed. For example, a list of application forms with links is displayed, such as "Moving out notification," "Moving in notification," "Moving address notification," "Marriage notification," "Birth notification," "Child allowance application," etc.

[0043] A link 806 for returning to the menu is a link that, when clicked, transitions to a higher-level menu screen than the screen 800 shown in FIG. 8, such a screen. The logout link 807 is a link that, when clicked, logs out the currently logged-in resident. If a resident is not logged in, for example, a login link is displayed instead of the logout link 807, and when clicked, a login screen is displayed where the user is prompted to enter their ID and password, and if the correct information is entered, the resident can log in.

[0044] [Question and answer flow for the digital customer service system] FIG. 9 is a flowchart showing the flow of operations for questions and answers from the user terminal 103 to the digital window server 101. First, a resident using the digital window server 101 logs in using the user terminal 103 to begin use. There are various possible methods for logging in, such as inputting an ID and password, authentication using a My Number card, etc. In this login, the digital window server 101 obtains information about the logged-in resident from the resident information database 311 (step S11).

[0045] Next, the resident enters a question in text format into the question input area 801 and presses the question button 802 to input the question into the system (step S12). The input question 911 is, for example, "I'm moving to Ward A." Next, the digital window server 101 displays the input question in the chat display area 803 (step S13).

[0046] When a question is input, the question and answer processing unit 302 generates a response using the input question and the acquired information about the resident (step S14). When generating this response, question and answer processing unit 302 generates the response using the model selected in advance by model evaluation unit 305. Response 912 may be created as, for example, "Please submit a notification of moving out."

[0047] Next, the digital window server 101 displays the generated response in the chat display area 803 on the screen 80 of the user terminal 103 (step S15). The resident using the system checks the answer displayed in the chat display area 803 and, if necessary, enters the next question (step S16). In this way, the resident continues to use the digital window server 101. When the resident has finished using the digital window system, he or she logs out.

[0048] [Model evaluation process flow] FIG. 10 is a flowchart showing the flow of processing by the model evaluation unit 305. The evaluation process in the model evaluation unit 305 can be executed by an administrator of the digital window server 101 or the like starting a command line or a batch file. Alternatively, the evaluation process can be started by the administrator or the like operating an administrator screen separately prepared by the digital window server 101.

[0049] When the process in the model evaluation unit 305 starts, the model evaluation unit 305 executes a process of collecting the correctness of the answer for model evaluation by the model evaluation answer correctness collecting unit 306 (step S21). The model evaluation answer correctness collecting unit 306 receives the model database 314 and the model evaluation question list 315 as input, acquires the correctness of the answers to each question of each model, and stores the data in the model answer correctness data 316. Details of the processing of the model evaluation answer correctness collecting unit 306 will be described later with reference to FIG.

[0050] Next, the item response theory calculation unit 307 performs item response theory calculations using the model answer correct / incorrect data 316 as input (IRT calculation processing), and outputs the model evaluation results 317 (step S22). Next, the model selection unit 308 receives the model evaluation value data as an input and selects the selected model 1002 (step S23). The selected model 1002 becomes the model used by the question and answer processing unit 302 thereafter. As a method for selecting a model, the model selection unit 308 selects the model with the highest evaluation value calculated by the item response theory calculation unit 307. Alternatively, the model selection unit 308 may use other methods, such as a method in which an administrator selects a model after looking at the results of the model calculated by the item response theory calculation unit 307, or a method in which an administrator performs an operation or the like to correct the evaluation results before selecting a model. When the selection process in the model selection unit 308 is completed, the model evaluation process ends.

[0051] [Process flow for collecting correct answers for model evaluation] FIG. 11 is a flowchart showing the process of collecting correct / incorrect answers for model evaluation by the model evaluation answer correct / incorrect collecting unit 306. When the process starts, the model evaluation answer correctness collecting unit 306 selects models to be evaluated from the model database 314 in order (step S31). Next, the model evaluation answer correctness collecting unit 306 selects questions in order from the model evaluation question list 315 (step S32).

[0052] Thereafter, the model evaluation answer correct / incorrect collection unit 306 acquires the answer to the selected question of the selected model (step S33). Then, the model evaluation answer correctness collecting unit 306 judges whether the acquired answer is correct or incorrect (step S34), and records the judgment result in the model answer correctness data 316 (step S35). Next, the model evaluation answer correct / incorrect collecting unit 306 determines whether or not all questions have been selected for the selected model (step S36). If all questions have not been selected in step S36 (No in step S36), the model evaluation answer correct / incorrect collecting unit 306 returns to step S32, selects the next question, and continues the process.

[0053] If all questions have been selected in step S36 (Yes in step S36), the model evaluation answer correct / incorrect collecting unit 306 determines whether all models have been selected (step S37). If all models have not been selected in step S37 (No in step S37), the model evaluation answer correct / incorrect collection unit 306 returns to step S31, selects the next AI model, and continues the process. If all models have been selected in step S37 (Yes in step S37), the model evaluation answer correctness collecting unit 306 ends the model evaluation answer correctness collecting process.

[0054] As described above, the digital window system of this embodiment applies item response theory calculations to the correct / incorrect answer data to calculate the evaluation value of each model, checks the evaluation value, selects one model, and uses the selected model for subsequent question and answer sessions. This allows the digital window system of this embodiment to generate answers to questions using the highest quality AI model and provide them to residents.

[0055] <Second embodiment> Next, a service providing system, a service providing method, and a program according to a second embodiment of the present invention will be described with reference to Figures 12 and 13. In Figures 12 and 13, parts corresponding to those in Figures 1 to 11 described in the first embodiment are given the same reference numerals, and duplicated explanations will be omitted.

[0056] Like the first embodiment, the second embodiment is also a service provision system provided by a government office to residents, and the government office is configured as a digital counter system that provides resident services and accepts applications from residents using information processing terminals. The basic configuration and processing of the service provision system are the same as those of the service provision system described in the first embodiment.

[0057] In the second embodiment, the digital window system of the first embodiment has already performed model evaluation at least once. In this embodiment, if model evaluation results exist, the evaluation values ​​of each existing question are used when evaluating an additional model, thereby reducing the time and calculation costs required for model evaluation. In other words, the auxiliary storage device 206 shown in FIG. 3 includes the additional model evaluation unit 1200, which will be described next.

[0058] [Additional model evaluation process] FIG. 12 is a flowchart showing the flow of processing in the additional model evaluation unit 1200. The method of executing the additional model evaluation function is the same as the evaluation by the model evaluation unit 305 in the first embodiment. However, it is assumed that the function is started by explicitly specifying that an additional model will be evaluated using the existing evaluation results. The processing of the additional model evaluation unit 1200 will be explained with reference to FIG. 12. When the additional model evaluation unit 1200 starts the additional model evaluation processing, it executes the process of collecting correct / incorrect answers for additional model evaluation (step S41).

[0059] In the process of collecting correct answers for additional model evaluation, the additional model evaluation unit 1200 receives the model 1210 to be newly evaluated and the model evaluation question list 315 as input, obtains the correct answers to each question of the newly evaluated model 1210, and stores the correct answers in the model answer correctness data 316. The process of collecting correct answers for additional model evaluation will be described in detail later with reference to FIG. Next, the item response theory calculation unit 307 executes an item response theory calculation process when using existing question evaluation values ​​(step S42). That is, the item response theory calculation unit 307 performs item response theory calculations using the correctness or incorrectness of the answers of the new evaluation model in the model answer correctness data 316 and the existing question evaluation value data 700 as input, and stores the evaluation results of the new model in the model evaluation value data 600. The subsequent processing is the same as that of the model evaluation unit 305 in the first embodiment, and the model selection unit 308 performs model selection processing based on the model evaluation value data 600 (step S43), obtains the selected model 1002, and terminates the processing.

[0060] [Additional model evaluation answer correctness collection process] FIG. 13 is a flowchart showing the process of collecting correct / incorrect answers for evaluating additional models, which is executed in step S41 of FIG. First, when the additional model evaluation answer correctness collecting unit 1300 starts the additional model evaluation answer correctness collecting process, it selects the model 1210 to be newly evaluated (step S51). Next, the additional model evaluation answer correct / incorrect collecting unit 1300 selects questions in order from the model evaluation question list 315 (step S52).

[0061] Next, the additional model evaluation answer correct / incorrect collection unit 1300 acquires the answer to the selected question of the selected model (step S53). Furthermore, the additional model evaluation answer correctness collecting unit 1300 judges whether the acquired answer is correct or incorrect (step S54), and records the judgment result in the model answer correctness data 316 (step S55). Thereafter, the additional model evaluation answer correct / incorrect collection unit 1300 determines whether or not all questions have been selected for the selected model (step S56).

[0062] If all questions have not been selected in step S56 (No in step S56), the additional model evaluation answer correct / incorrect collection unit 1300 returns to the process in step S52, selects the next question, and continues the process. If all questions have been selected in step S56 (Yes in step S56), the additional model evaluation answer correct / incorrect collection unit 1300 ends the process.

[0063] As described above, in the digital counter system of this embodiment, when evaluating a new model to be evaluated, the time and computational costs required for evaluating the model can be reduced by utilizing existing question evaluation value data.

[0064] <Modification> The embodiment examples described so far have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described.

[0065] For example, in the digital counter system of each embodiment, residents and administrators can check the contents of the evaluation of models and questions, and provide support to help residents gain consent for the use of AI, or to help administrators manage models and questions. 14 shows an example of a model evaluation confirmation screen 1400 in this case. The model evaluation confirmation screen 1400 is displayed on the administrator terminal when used by the administrator, and is displayed on the user terminal 103 (FIG. 1) when used by a resident.

[0066] The model evaluation confirmation screen 1400 is a screen for residents to check the evaluation values ​​of the model and questions, and to confirm the quality of the model and the accuracy of the answers to the questions, in order to gain a sense of satisfaction with the currently selected model. A possible method for transitioning to the model evaluation confirmation screen 1400 is to provide a link on the digital window system screen 800 or on another screen such as a menu.

[0067] The model evaluation confirmation screen 1400 includes a table 1401 of the evaluation value of the question and the correctness of the model, an evaluation value 1402 of the model, a confirmation button 1403, and the like. The table 1401 of question evaluation values ​​and model correctness / incorrectness displays, for each question, the content, resident attributes, correct answer, difficulty level, discriminatory power, and correctness / incorrectness of the answer of the currently selected model as a table. The model evaluation value 1402 displays the evaluation value of the currently selected model evaluated by the item response theory. Furthermore, such a model evaluation confirmation screen 1400 may be viewed by a system administrator or the like, and may be used to select a model or modify the model evaluation question list. In this way, residents and administrators can check the evaluations of models and questions, allowing administrators and users (residents) to select more appropriate models and provide desirable services.

[0068] Furthermore, in each of the above-mentioned embodiments, the present invention was applied to model selection processing when a government office's digital counter system uses generative AI technology, but the present invention can be applied to various technologies that use models such as generative AI technology and classification AI technology.

[0069] 1 to 3, only control lines and information lines that are considered necessary for explanation are shown, and not all control lines and information lines are necessarily shown in the product. In reality, it can be considered that almost all components are interconnected. Furthermore, the digital window server 101 shown in FIG. 1 may perform some of the processing in a separate device, rather than performing all processing in a single device.

[0070] For example, the model (selected model 1002) executed by the digital window server 101 may be a model stored in an external server connected via the network 102, and the digital window server 101 may access the external server to evaluate the model. Furthermore, the flowcharts shown in FIGS. 9 to 13 are also examples, and as long as the processing results are the same, the order of some of the processing may be changed or multiple processes may be executed simultaneously.

[0071] Furthermore, the digital window server 101 described in each of the above-mentioned embodiments may be configured by implementing a program that executes the processes shown in flowcharts such as Figures 9 to 13, and in this case the program is prepared in the main storage device 202 or auxiliary storage device 206 in the computer shown in Figure 2. Alternatively, the program executed by the computer shown in Figure 2 may be stored in a recording medium such as an external memory, IC card, SD card, or optical disk, and transferred to a computer that functions as the digital window server 101. [Explanation of symbols]

[0072] 101...Digital window server 102…Network 103...User terminal 201...CPU 202…Main memory 204...Display device 205...Input device 206…Auxiliary storage device 301...Resident Access Processing Unit 302...Question and answer processing section 303...Service provision processing unit 304...Application reception processing unit 305...Model Evaluation Department 306…Model evaluation answer correctness collection section 307...Item Response Theory Calculation Unit (IRT Calculation Unit) 308...Model selection section 311…Resident Information Database 312…Resident Services 313...Application Form Format 314...Model Database 315...Model Evaluation Questions List 316...Model answer correct / incorrect data 317...Model evaluation results 800...screen 801...Question input area 802...Question button 803...Chat display area 804…Resident service selection area 805...Application form selection area 1200…Additional Model Evaluation Section 1210...Newly evaluated model 1300…Additional model evaluation answer correct / incorrect collection section 1400...Model evaluation confirmation screen

Claims

1. a model evaluation answer correctness collecting unit that inputs a plurality of registered questions for each of the prepared models and collects the correctness of the answers to the output questions; an item response theory calculation unit that performs calculation processing based on item response theory on the correct / incorrect data collected by the model evaluation answer correct / incorrect collection unit to calculate evaluation values ​​of the plurality of models; a model selection unit that selects a model to be applied when providing a service based on the evaluation value calculated by the item response theory calculation unit. Service delivery system.

2. The item response theory calculation unit calculates and stores evaluation values ​​for the plurality of questions when calculating evaluation values ​​for the plurality of models. The service providing system according to claim 1 .

3. and a service providing unit that generates an answer to a question entered by a user in a service based on the model selected by the model selecting unit and displays the generated answer on a predetermined terminal. The service providing system according to claim 1 .

4. Further, an additional model evaluation unit is provided, The additional model evaluation unit performs an item response theory calculation using existing question evaluation values ​​to obtain an evaluation value for a model to be newly evaluated. The service providing system according to claim 1 .

5. The service providing unit displays, on the terminal, evaluation values ​​of the plurality of models, contents of registered questions, whether answers of the plurality of models to the registered questions are correct or incorrect, and evaluation values ​​of the plurality of questions. The service providing system according to claim 3 .

6. A service providing method in which, when a computer provides a service by calculation using a model, the computer selects the model through calculation processing, The computational processing by the computer includes: a model evaluation answer correctness collection process for inputting a plurality of registered questions for each of the prepared models and collecting the correctness of the answers to the output questions; an item response theory calculation process that performs a calculation process based on item response theory on the correct / incorrect data collected by the model evaluation answer correct / incorrect collection process to calculate evaluation values ​​of the plurality of models; and a model selection process for selecting a model to be applied when providing a service based on the evaluation value calculated by the item response theory calculation process. Service delivery methods.

7. a model evaluation answer correctness collection procedure for inputting multiple registered questions for each of the multiple prepared models and collecting the correctness of the answers to the output questions; an item response theory calculation procedure for calculating evaluation values ​​of the plurality of models by performing calculation processing based on item response theory on the correct / incorrect data collected by the model evaluation answer correct / incorrect collection procedure; a model selection step of selecting a model to be applied when providing a service based on the evaluation value calculated by the item response theory calculation step. program.

Images