Evaluation device and evaluation program

Abstract

To reflect information on each entity which has provided original data for learning, in evaluation.SOLUTION: An evaluation apparatus 100 includes: a learned model storage unit 12 which stores a plurality of learned models corresponding to each of a plurality of unique determination entities; an evaluation target data acquisition unit 14 which acquires evaluation target data to be subjected to evaluation; and an evaluation execution unit 16 which evaluates the evaluation target data acquired by the evaluation target data acquisition unit 14 using the learned models stored in the learned model storage unit 12. Each of the learned models is obtained by learning multiple pieces of unique learning data which are obtained by determinations made by unique determination entities corresponding to the learned models, for each of multiple pieces of determination data in a specific theme.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an evaluation device and an evaluation program. [Background technology]

[0002] There are known devices that use so-called artificial intelligence to evaluate objects. For example, Patent Document 1 discloses a device that evaluates clothes suitable for a specified location and suggests them to a user. This device performs an Internet search to collect images of the location, and evaluates the clothes using an artificial intelligence system constructed by learning from the collected images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107084 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned device, regardless of the source of each image that was the original data for learning, a single artificial intelligence system is constructed by collecting images collected from various Internet sites. For this reason, for example, information about each subject providing the images (such as information about the individual preferences of each subject) tends to be buried, making it easier to make average evaluations.

[0005] Therefore, an object of the evaluation device and evaluation program according to the present disclosure is to enable evaluation that reflects information about each subject that provided the original data for learning. [Means for solving the problem]

[0006] An evaluation device (100) according to one embodiment of the present disclosure includes a trained model memory unit (12) that stores multiple trained models (N) corresponding to multiple unique judgment entities (S), an evaluation target data acquisition unit (14) that acquires evaluation target data (ED) to be evaluated, and an evaluation execution unit (16) that uses the multiple trained models (N) stored in the trained model memory unit (12) to evaluate the evaluation target data (ED) acquired by the evaluation target data acquisition unit (14), and each of the multiple trained models (N) is a model trained from multiple unique training data (TD) that is the result of the unique judgment entity (S) corresponding to the trained model (N) making a judgment on each of multiple judgment data (JD) on a specific theme.

[0007] According to this evaluation device (100), a trained model (N) is a model trained on a plurality of unique learning data (TD) that is the result of the unique judgment subject (S) making a judgment on each of a plurality of judgment data (JD), and is stored for each of the plurality of unique judgment subjects (S). Then, evaluation of the evaluation target data (ED) is performed using these plurality of trained models (N). Therefore, this evaluation device (100) can make an evaluation that reflects information for each subject that provided the original data for learning.

[0008] In the evaluation device (100) according to an embodiment of the present disclosure, each of the plurality of judgment data (JD) may be data including multiple-choice questions, which makes it easier to statistically process the judgment results of each judgment data (JD) by the unique judgment entity (S).

[0009] In an evaluation device (100) according to an embodiment of the present disclosure, each of the multiple trained models (N) may be a model trained on 30 or more unique training data (TD) that are the results of a unique judgment entity (S) corresponding to the trained model (N) making a judgment on each of 30 or more pieces of judgment data (JD). This can improve the accuracy of evaluation of the evaluation target data (ED) using the trained model (N).

[0010] In the evaluation device (100) according to an aspect of the present disclosure, the number of options for a question may be 2. This makes it easier for the unique judge (S) to judge the judgement data (JD) compared to a case where the number of options is 3 or more.

[0011] In the evaluation device (100) according to one embodiment of the present disclosure, the number of unique learning data groups (TDG) consisting of multiple unique learning data (TD) that are the results of multiple unique judgment entities (S) making judgments on multiple judgment data (JD) may be the same as the number of unique judgment entities (S) and may also be the same as the number of trained models (N). This makes it possible to make an evaluation that more accurately reflects information about each entity that provided the original data for learning.

[0012] In the evaluation device (100) according to one embodiment of the present disclosure, each of the multiple unique learning data groups (TDG) may be in one-to-one correspondence with any unique decision subject (S) and may also be in one-to-one correspondence with any trained model (N). This enables evaluation that more accurately reflects information about each subject that provided the original data for learning.

[0013] The evaluation device (100) according to one embodiment of the present disclosure includes an attribute information acquisition unit (13) that acquires attribute information indicating the attributes of each of a plurality of unique judgment subjects (S), and a target attribute selection unit (15) that selects a target attribute that is a target attribute. The evaluation execution unit (16) may extract, from the plurality of unique judgment subjects (S), a unique judgment subject (S) having a target attribute selected by the target attribute selection unit (15) based on the attribute information acquired by the attribute information acquisition unit (13), and evaluate the evaluation target data (ED) using a trained model (N) corresponding to the extracted unique judgment subject (S). This makes it possible to retroactively acquire the attribute of the unique judgment subject (S) corresponding to the trained model (N) used in the evaluation from the result of the evaluation. In addition, the evaluation target data (ED) can be evaluated using a trained model (N) corresponding to a unique judgment subject (S) with a more suitable attribute, for example, according to the user's request or the contents of the evaluation target data (ED).

[0014] In the evaluation device (100) according to one embodiment of the present disclosure, the evaluation execution unit (16) may perform a first-stage evaluation of the multiple evaluation target data (ED) using each of the multiple trained models (N) to obtain multiple first-stage evaluation results, and perform a second-stage evaluation of the multiple evaluation target data (ED) based on the obtained multiple first-stage evaluation results to obtain a second-stage evaluation result, thereby evaluating the multiple evaluation target data (ED). According to this, for example, a composite evaluation (i.e., a first-stage evaluation) that takes into account multiple types of elements is performed by each of the multiple trained models (N) for the evaluation target data (ED) that includes multiple types of elements. Then, the evaluation of the evaluation target data (ED) (i.e., a second-stage evaluation) is performed by combining the multiple evaluation results (i.e., the multiple first-stage evaluation results) that take into account these multiple types of elements. Therefore, the evaluation of the entire evaluation target data (ED) can be performed using an evaluation that takes into account the elements of the evaluation target data (ED) that includes multiple types of elements, thereby improving the accuracy of the evaluation.

[0015] In the evaluation device (100) according to one embodiment of the present disclosure, the evaluation execution unit (16) may evaluate the evaluation target data (ED) by obtaining multiple unique evaluation results (IR) by performing individual evaluation of the evaluation target data (ED) using each of the multiple trained models (N), and obtaining a comprehensive evaluation result (CR) by performing a comprehensive evaluation of the evaluation target data (ED) based on the multiple acquired unique evaluation results (IR), thereby evaluating the evaluation target data (ED). According to this, it is possible to evaluate the evaluation target data (ED) after weighting each trained model (N) corresponding to each unique decision entity (S) according to, for example, a user's request or the contents of the evaluation target data (ED).

[0016] The evaluation device (100) according to one embodiment of the present disclosure may include an attribute information acquisition unit (13) that acquires attribute information indicating the attributes of each of the multiple unique judgment subjects (S), and an evaluation result output unit (17) that outputs to the outside the attribute information of each of the multiple unique judgment subjects (S) corresponding to each of the multiple trained models (N) used in the individual evaluation. This allows the user to know information about the subject that provided the original data for learning in the trained model (N).

[0017] The evaluation device (100) according to one embodiment of the present disclosure may include an evaluation result output unit (17) that outputs to the outside a plurality of judgment data (JD) or a plurality of unique learning data (TD) corresponding to each of a plurality of trained models (N) used in the individual evaluation. This allows the user to learn information about the entity that provided the original data for learning in the trained model (N) by tracing back from the evaluation result of the evaluation target data (ED) to the judgment data (JD) or the unique learning data (TD). As a result, it becomes possible to learn information about the basis of the evaluation, and also to obtain suggestions for how to improve the evaluation target data (ED).

[0018] The evaluation device (100) according to one embodiment of the present disclosure includes a billing processing unit (18) that executes processing of billing related to the evaluation performed by the evaluation performing unit (16), and the amount of billing may be determined based on conditions related to the multiple trained models (N) used in the evaluation by the evaluation performing unit (16). This allows a more appropriate amount of billing to be charged to the user.

[0019] In the evaluation device (100) according to one aspect of the present disclosure, the conditions may include the number of multiple trained models (N) used by the evaluation execution unit (16). This makes it possible to charge the user an amount according to the number of multiple trained models (N) used by the evaluation execution unit (16).

[0020] The evaluation program according to one embodiment of the present disclosure may cause a computer to execute processing to evaluate evaluation target data (ED) using a plurality of trained models (N) that are models trained with a plurality of unique learning data (TD) that are the results of a plurality of unique judgment subjects (S) making judgments on a plurality of judgment data (JD) on a specific theme. In other words, the evaluation program may be a program for causing a computer to function as the evaluation device (100) according to any of the above-mentioned embodiments. This makes it possible to perform an evaluation that reflects information about each subject that provided the original data for learning.

[0021] Note that the reference symbols in parentheses above are intended to indicate the reference symbols of components in the embodiments described below as an example of the present disclosure, and are not intended to limit the present disclosure to the aspects of the embodiments. Effect of the Invention

[0022] In this way, the evaluation device and evaluation program according to the present disclosure can enable evaluation that reflects information about each subject that provided the original learning data. [Brief description of the drawings]

[0023] [Figure 1]FIG. 1 is a block diagram showing a functional configuration of an evaluation device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a determination data group. [Diagram 3] FIG. 3 is a diagram for explaining that each trained model corresponds to each unique decision-making entity. [Figure 4] FIG. 4 is a diagram for explaining that evaluation of evaluation target data is performed using multiple trained models. [Diagram 5] FIG. 5 is a sequence diagram showing a learning process for generating a trained model. [Figure 6] FIG. 6 is a sequence diagram showing an evaluation process for evaluating evaluation target data. [Figure 7] FIG. 7 is a sequence diagram showing a charging process for performing charging. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, exemplary embodiments will be described with reference to the drawings. In addition, the same or corresponding parts in each drawing are given the same reference numerals, and duplicated explanations will be omitted.

[0025] [First embodiment] [Evaluation device] A first embodiment will be described. FIG. 1 is a block diagram showing a functional configuration of an evaluation device 100 according to an embodiment. FIG. 2 is a diagram showing an example of a judgment data group JDG. FIG. 3 is a diagram for explaining that each trained model N corresponds to each unique judgment subject S. FIG. 4 is a diagram for explaining that the evaluation target data ED is evaluated using a plurality of trained models N. As shown in FIGS. 1 to 4, the evaluation device 100 according to the first embodiment is a device that evaluates an object using so-called artificial intelligence (AI (Artificial Intelligence)), and specifically, is a device that evaluates the evaluation target data ED using a plurality of trained models N corresponding to each of a plurality of unique judgment subjects S. Each trained model N is a model that has been trained with a plurality of unique learning data TD, which is a result of the unique judgment subject S corresponding to the trained model N making a judgment on each of a plurality of judgment data JD on a specific theme. The evaluation target data ED is data to be evaluated, and is data on the same theme as the theme of the judgment data JD.

[0026] Here, the evaluation device 100 evaluates photos of other golf courses as evaluation target data ED using multiple trained models N, which are models that have been trained to learn the golf course photo preferences of each unique decision entity S. Note that what the evaluation device 100 evaluates is not limited to golf course photos, and may be, for example, website designs (designs of portal sites, banners, advertisements, etc.), images, or copywriting, or products or services on electronic commerce (EC (Electronic Commerce)) sites, etc. Alternatively, what the evaluation device 100 evaluates may be content or posts on SNS (Social Networking Service), video services, flea market apps, etc. Note that the evaluation viewpoints for websites are not particularly limited, and may be, for example, the ease of understanding of the page transition flow (so-called site flow), the reliability of the site, or the likeness of a corporate site (i.e., the degree to which a user who views the site can easily recognize the site as a corporate site), etc.

[0027] The evaluation device 100 uses deep learning using a neural network as artificial intelligence. In other words, a "trained model" is a neural network trained with a plurality (a large number) of teacher data, and here, a plurality of unique learning data TD are the teacher data. These multiple unique learning data TD are also collectively referred to as a unique learning data group TDG. The trained model N includes an input layer, an intermediate layer (hidden layer), and an output layer, and when evaluation target data ED is input to the input layer, an evaluation result is output from the output layer. Note that the artificial intelligence method used by the evaluation device 100 is not limited to deep learning, and may be various types of machine learning.

[0028] The "unique judgment subject" is a subject that makes a judgment on each judgment data JD, for example, an individual. Here, a woman in her twenties, a man in his twenties, and a woman in her eighties are exemplified as multiple unique judgment subjects S. Note that the unique judgment subject S is not limited to an individual, and may be an animal or a robot (an artificial intelligence separate from the evaluation device 100), etc.

[0029] "Data for judgment" is data judged by a unique judgment subject S. Specifically, the data for judgment JD is data related to a specific theme, and a plurality of pieces are collected and used. These plurality of data for judgment JD are also collectively referred to as a group of data for judgment JDG. The group of data for judgment JDG may be, for example, a questionnaire administered to the unique judgment subject S, who is an individual. Here, each data for judgment JD is data including a multiple-choice question, and the number of options for this question is two. More specifically, here, each data for judgment JD is data including a photo of a golf course and a question with two options (two options) of whether the photo is liked or disliked. The content of the two options is not limited to like or dislike, and can be any content such as A or B, OK or NG. The number of options may be three or more, and as an example, a seven-point evaluation using seven options may be performed.

[0030] In this way, the unique learning data TD is data in which data on the judgment result by the unique judgment subject S is added to each judgment data JD. Here, the unique learning data TD is data in which information on whether the unique judgment subject S likes or dislikes the photo of a golf course, which is the judgment data JD, is added (in FIG. 3, the case where it is liked is indicated by a circle, and the case where it is not liked is indicated by a cross).

[0031] Each trained model N is a model trained with a plurality of unique learning data TD, which are the results of the unique judgment entity S corresponding to the trained model N making a judgment on each judgment data JD. Here, the number of judgment data JD is 12. In other words, each trained model N is a model trained with 12 unique learning data TD, which are the results of the unique judgment entity S corresponding to the trained model N making a judgment on each of the 12 judgment data JD. Preferably, the number of judgment data JD may be 30 or more. For example, when the number of judgment data JD is 30 or more, each trained model N is a model trained with 30 or more unique learning data TD, which are the results of the unique judgment entity S corresponding to the trained model N making a judgment on each of the 30 or more judgment data JD. Here, the number of unique learning data TD may be the same as the number of judgment data JD. More preferably, the number of judgment data JD may be 100 or more, or may be 300 or more.

[0032] Here, the number of unique learning data groups TDG consisting of multiple unique learning data TD, which are the results of each unique judgment entity S making a judgment on each judgment data JD, is the same as the number of unique judgment entities S and the same as the number of trained models N. More specifically, each unique learning data group TDG is in one-to-one correspondence with any unique judgment entity S and is also in one-to-one correspondence with any trained model N. In other words, the unique learning data group TDG, the unique judgment entity S, and the trained model N are in one-to-one correspondence.

[0033] The evaluation device 100 is configured as a server (computer) capable of wireless or wired communication with various devices such as the judgment subject terminal 200, the user terminal 300, and the billing server 400. The evaluation device 100 includes, as physical components, a control arithmetic device, a storage device, and an input / output device. The control arithmetic device is configured by a controller 1 such as a CPU (Central Processing Unit), and executes arithmetic processing and controls the storage device and the input / output device. The storage device includes, for example, a main storage device and an auxiliary storage device. The main storage device includes, for example, a RAM (Random Access Memory). The auxiliary storage device includes, for example, a ROM (Read Only Memory). The input / output device includes, for example, an input device that receives data from the outside and transmits it to the storage device, and an output device that outputs to the outside the arithmetic results that are calculated by the control arithmetic unit and stored in the storage device. The evaluation device 100 executes a predetermined process by, for example, loading a program stored in the ROM into the RAM and executing the program loaded into the RAM with the CPU. The evaluation device 100 may be configured as a computer, and may have a configuration different from the above-mentioned configuration.

[0034] The controller 1 of the evaluation device 100 functionally comprises a judgment data presentation unit 10, a learning processing unit 11, a learned model memory unit 12, an attribute information acquisition unit 13, an evaluation target data acquisition unit 14, a target attribute selection unit 15, an evaluation execution unit 16, an evaluation result output unit 17, and a billing processing unit 18.

[0035] The judgment data presenting unit 10 presents the judgment data group JDG (each judgment data JD) to the judgment subject terminal 200 (details of the judgment subject terminal 200 will be described later). Specifically, the judgment data presenting unit 10 acquires the judgment data group JDG, for example, from an external source, stores it, and transmits it to the judgment subject terminal 200.

[0036] The learning processing unit 11 receives the unique learning data group TDG from the judgment entity terminal 200, and generates a trained model N corresponding to the unique judgment entity S related to the unique learning data group TDG based on the received unique learning data group TDG. Specifically, the learning processing unit 11 trains a neural network with multiple unique learning data TD, which are the results of the unique judgment entity S making a judgment on each judgment data JD, to generate the trained model N.

[0037] The trained model storage unit 12 acquires the trained model N generated by the learning processing unit 11. The trained model storage unit 12 stores a plurality of trained models N corresponding to each unique decision subject S.

[0038] The attribute information acquisition unit 13 acquires attribute information indicating the attributes of each unique judgment subject S. Specifically, the attribute information acquisition unit 13 receives attribute information of the unique judgment subject S linked to each unique learning data group TDG from the judgment subject terminal 200, and stores the attribute information linked to the unique judgment subject S and the unique learning data group TDG. The "attribute information" is information indicating the characteristics of the unique judgment subject S. Specifically, the attribute information may be information such as age (20s, 30s, etc.), sex, occupation, annual income, family structure, birthplace, residence, and place of work. Here, the attribute information acquisition unit 13 acquires attribute information of the three unique judgment subjects S, that is, the first unique judgment subject S is a woman in her 20s, the second unique judgment subject S is a man in his 20s, and the third unique judgment subject S is a woman in her 80s.

[0039] The evaluation target data acquisition unit 14 acquires evaluation target data ED to be evaluated. For example, the evaluation target data acquisition unit 14 receives evaluation target data ED that a user wishes to be evaluated by the evaluation device 100 from a user terminal 300 (the user terminal 300 will be described in detail later).

[0040] The target attribute selection unit 15 selects a target attribute, which is an attribute to be targeted. The "target attribute" means, for example, an attribute to be targeted for evaluation of the evaluation target data ED. The target attribute selection unit 15 may receive, from the user terminal 300, a target attribute of the unique judgment subject S corresponding to the trained model N that the user wishes to have evaluated by the evaluation device 100. Alternatively, the target attribute selection unit 15 may automatically select a target attribute based on a predetermined condition. The predetermined condition may be a condition related to the theme of the evaluation target data ED.

[0041] The evaluation execution unit 16 evaluates the evaluation target data ED acquired by the evaluation target data acquisition unit 14, using a plurality of trained models N stored in the trained model storage unit 12. Specifically, the evaluation execution unit 16 inputs the evaluation target data ED to the input layer of each trained model N, and outputs the evaluation result from the output layer.

[0042] The evaluation execution unit 16 may extract, from among the multiple unique judgment subjects S, a unique judgment subject S having a target attribute selected by the target attribute selection unit 15 based on the attribute information acquired by the attribute information acquisition unit 13, and evaluate the evaluation target data ED using a trained model N corresponding to the extracted unique judgment subject S. In other words, the evaluation execution unit 16 may evaluate the evaluation target data ED using only a portion of the trained models N stored in the trained model storage unit 12 (one or more trained models N corresponding to the unique judgment subject S having the selected target attribute).

[0043] In addition, the evaluation execution unit 16 may evaluate the evaluation target data ED by acquiring multiple unique evaluation results IR by performing individual evaluation of the evaluation target data ED using each trained model N, and acquiring a comprehensive evaluation result CR by performing a comprehensive evaluation of the evaluation target data ED based on the acquired unique evaluation results IR. For example, when multiple unique evaluation results IR are acquired, the evaluation execution unit 16 may calculate a comprehensive evaluation result CR by aggregating (for example, averaging, summing, or summing after weighting) the acquired multiple unique evaluation results IR. Alternatively, when a single unique evaluation result IR is acquired, the evaluation execution unit 16 may consider the acquired unique evaluation result IR as the comprehensive evaluation result CR. Here, "individual evaluation" means that the evaluation target data ED is evaluated (i.e., individual evaluation) using a single trained model N. Also, "comprehensive evaluation" means that the evaluation target data ED is evaluated (i.e., comprehensive evaluation) using multiple trained models N.

[0044] In Fig. 4, for the input evaluation target data ED, the unique evaluation result IR by the trained model N corresponding to a woman in her twenties (unique decision-making subject S) is a cross (does not like), the unique evaluation result IR by the trained model N corresponding to a man in his twenties (unique decision-making subject S) is a circle (likes), and the unique evaluation result IR by the trained model N corresponding to a woman in her eighties (unique decision-making subject S) is a circle (likes). The overall evaluation result CR calculated based on these unique evaluation results IR is a B rating in the three-level evaluation of ABC. Note that such an evaluation method for the unique evaluation result IR and the overall evaluation result CR is merely an example, and another evaluation method may be adopted.

[0045] Returning to FIG. 1 to FIG. 4, the evaluation result output unit 17 outputs information on the evaluation result of the evaluation target data ED output by the evaluation execution unit 16 (evaluation result information) to the outside. In addition, the evaluation result output unit 17 outputs attribute information of each unique judgment subject S corresponding to each trained model N used in the individual evaluation of the evaluation target data ED by the evaluation execution unit 16 to the outside. For example, the evaluation result output unit 17 may output at least one of the evaluation result information and the attribute information to the user terminal 300 as information to be displayed on the display 4 in the user terminal 300. Alternatively, the evaluation result output unit 17 may output at least one of the evaluation result information and the attribute information to the storage device of the evaluation device 100 or the user terminal 300 as information to be stored in the storage device such as the evaluation device 100 or the user terminal 300. The evaluation result output unit 17 may output information to be displayed in the form of a table or a graph to the outside.

[0046] The evaluation result output unit 17 may output information combining the evaluation result information and the attribute information to the outside. For example, the evaluation result output unit 17 may aggregate information on the results of an evaluation (individual evaluation) using a trained model N corresponding to an individual decision-making entity S having a specific attribute for each of a plurality of attributes and output the aggregated information to the outside. As an example, the evaluation result output unit 17 may output evaluation result information aggregated for each of a first attribute (e.g., an individual decision-making entity S in their 20s) and a second attribute different from the first attribute (e.g., an individual decision-making entity S in their 80s) to the outside.

[0047] Furthermore, the evaluation result output unit 17 may output to the outside a plurality of judgment data JD or a plurality of unique learning data TD corresponding to each of the plurality of trained models N used in the individual evaluation. For example, the evaluation result output unit 17 may output to the outside at least one of the plurality of judgment data JD and the plurality of unique learning data TD used when each of the plurality of trained models N used in the individual evaluation is generated by the learning processing unit 11.

[0048] The charging processing unit 18 executes a charging process related to the evaluation performed by the evaluation execution unit 16. The charging processing unit 18 outputs charging amount information indicating the amount of the charge to the user terminal 300. At this time, the charging processing unit 18 may output breakdown information indicating the breakdown of the amount of the charge to the user terminal 300 together with the charging amount information. In response to outputting the charging amount information to the user terminal 300, the charging processing unit 18 accepts charging instruction information indicating that the user of the user terminal 300 has accepted the charging. When the charging processing unit 18 accepts the charging instruction information from the user terminal 300, the charging processing unit 18 executes a charging procedure in cooperation with the charging server 400 (the charging server 400 will be described in detail later). When the charging procedure with the charging server 400 is completed, the charging processing unit 18 outputs a charging completion notification notifying the user terminal 300 that charging has been completed.

[0049] Specifically, as a procedure for charging, charging processor 18 transmits data (charging processing data) including the amount of charging and the user's consent to charging server 400. After that, when charging server 400 completes charging, charging completion information indicating that charging has been completed is received from charging server 400. Note that the specific contents of the information included in the charging processing data and the charging completion information are not limited to those described above. In addition, the specific method of the charging procedure is not particularly limited, and various methods including, for example, known cashless payment, may be adopted.

[0050] The amount of the charge may be determined based on conditions related to the multiple trained models N used in the evaluation by the evaluation execution unit 16. This condition may include the number of the multiple trained models N used by the evaluation execution unit 16. This condition may also include the number (amount) of the multiple judgment data JD trained by each trained model N, or the attributes of the unique judgment subject S corresponding to the trained model N used to evaluate the evaluation target data ED.

[0051] As a more detailed example, for example, the evaluation target data ED may be evaluated by dividing the evaluation target (judgment data JD and evaluation target data ED) into a plurality of parts, evaluating each part (partial evaluation) to obtain a plurality of partial evaluation results, and evaluating the whole (overall evaluation) based on the obtained plurality of partial evaluation results to obtain an overall evaluation result. For example, if the evaluation target is a website, the website may be divided into parts such as a layout, an eye-catching image, and a catch copy, and the evaluation may be performed. In this case, some of the parts may be free of charge, and the remaining parts may be charged.

[0052] [Decision-making device] The judgment subject terminal 200 is a terminal used by the unique judgment subject S. The judgment subject terminal 200 includes, for example, an input device and an output device, and is configured to be capable of wireless or wired communication with another device such as the evaluation device 100. The judgment subject terminal 200 may be, for example, a smartphone, a tablet, a notebook computer, a desktop computer, etc., and a smartphone is exemplified as the judgment subject terminal 200 here.

[0053] The judgment subject terminal 200 includes, as its physical configuration, a control arithmetic device, a storage device, and an input / output device. The control arithmetic device is configured by a controller 2 such as a CPU, and executes arithmetic processing and controls the storage device and the input / output device. The storage device includes, for example, a main storage device and an auxiliary storage device. The main storage device includes, for example, a RAM. The auxiliary storage device includes, for example, a ROM. The input / output device includes, for example, an input device that receives data from the outside and transmits the data to the storage device, and an output device that outputs to the outside the results of calculations that are calculated by the control arithmetic unit and stored in the storage device. The judgment subject terminal 200 executes a predetermined process, for example, by loading a program stored in the ROM into the RAM and executing the program loaded into the RAM by the CPU.

[0054] The controller 2 of the decision-making terminal 200 functionally comprises a decision input unit 20 .

[0055] The judgment input unit 20 outputs the judgment data group JDG (each judgment data JD) presented by the judgment data presentation unit 10 of the evaluation device 100 on an output device such as a display, and accepts input of the judgment result of the unique judgment entity S on the judgment data group JDG. For example, the judgment input unit 20 accepts the judgment result of the unique judgment entity S via an input device of the judgment entity terminal 200. Then, the judgment input unit 20 adds data related to the judgment result by the unique judgment entity S to the judgment data group JDG (each judgment data JD) to generate a unique learning data group TDG (each unique learning data TD).

[0056] When receiving a result of a judgment made on the judgment data group JDG from the unique judgment entity S, the judgment input unit 20 may receive attribute information indicating the attributes of the unique judgment entity S. As a result, the judgment input unit 20 acquires the attribute information of the unique judgment entity S that made a judgment on the judgment data group JDG by linking it to the unique learning data group TDG by the unique judgment entity S.

[0057] The judgment subject terminal 200 transmits the acquired unique learning data group TDG and attribute information of the unique judgment subject S linked to the unique learning data group TDG to the evaluation device 100. The judgment subject terminal 200 may transmit the unique learning data group TDG and the attribute information of the unique judgment subject S to the evaluation device 100 upon generating the unique learning data group TDG.

[0058] [User device] The user terminal 300 is a terminal used by a user who is going to evaluate the evaluation target data ED. The user terminal 300 includes, for example, an input device and an output device, and is configured to be capable of wireless or wired communication with another device such as the user terminal 300. The user terminal 300 may be, for example, a smartphone, a tablet, a notebook computer, a desktop computer, or the like, and here, a smartphone is exemplified as the user terminal 300.

[0059] The user terminal 300 includes, as physical components, a control arithmetic unit, a storage device, and an input / output device. The control arithmetic unit is configured with a controller 3 such as a CPU, and executes arithmetic processing and controls the storage device and the input / output device. The storage device includes, for example, a main storage device and an auxiliary storage device. The main storage device includes, for example, a RAM. The auxiliary storage device includes, for example, a ROM. The input / output device includes, for example, an input device that receives data from the outside and transmits the data to the storage device, and an output device such as a display 4 that outputs to the outside the results of calculation performed by the control arithmetic unit and stored in the storage device. The user terminal 300 executes a predetermined process, for example, by loading a program stored in the ROM into the RAM and executing the program loaded into the RAM by the CPU.

[0060] The controller 3 of the user terminal 300 functionally comprises an evaluation object data input unit 30 , an object attribute input unit 31 , an output processing unit 32 , and a charging instruction unit 33 .

[0061] The evaluation target data input unit 30 accepts input of evaluation target data ED that the user wishes to be evaluated by the evaluation device 100. For example, the evaluation target data input unit 30 acquires the evaluation target data ED that the user inputs via the input device of the user terminal 300. Thereafter, the evaluation target data input unit 30 inputs the acquired evaluation target data ED to the evaluation target data acquisition unit 14 of the evaluation device 100.

[0062] The target attribute input unit 31 accepts input of an attribute (target attribute) of the unique judgment subject S corresponding to the trained model N that the user wishes to use when evaluating the evaluation device 100. For example, the target attribute input unit 31 acquires a target attribute that the user inputs via an input device of the user terminal 300. Thereafter, the target attribute input unit 31 inputs the acquired target attribute to the target attribute selection unit 15 of the evaluation device 100.

[0063] The output processing unit 32 acquires the evaluation result of the evaluation target data ED and attribute information (target attribute information) which is information on the attribute (target attribute) of the unique judgment subject S corresponding to the trained model N used to evaluate the evaluation target data ED from the evaluation result output unit 17 of the evaluation device 100. Then, the output processing unit 32 displays the acquired evaluation result and target attribute information on the display 4 of the user terminal 300.

[0064] The charging instruction unit 33 displays the charging amount information input from the charging processing unit 18 of the evaluation device 100 on the display 4 of the user terminal 300 to let the user know the amount of the charge. At this time, when breakdown information showing the breakdown of the charging amount is input from the charging processing unit 18 together with the charging amount information, the charging instruction unit 33 may also display the contents of the breakdown information on the display 4.

[0065] The charging instruction unit 33 confirms the user's intention to consent to charging. For example, the charging instruction unit 33 may display a charging consent button on the touch panel display 4 of the user terminal 300 and confirm the user's intention to consent to charging by touching the button. When the charging instruction unit 33 confirms the user's intention to consent to charging, it outputs charging instruction information that instructs the charging processing unit 18 of the evaluation device 100 to perform charging. The "charging instruction information" is information that commands the evaluation device 100 to execute the charging procedure in cooperation with the charging server 400.

[0066] When the billing procedure between the evaluation device 100 and the billing server 400 is completed, a billing completion notice notifying that billing has been completed is input from the billing processing unit 18 of the evaluation device 100 to the billing instruction unit 33. When the billing completion notice is input, the billing instruction unit 33 causes the display 4 of the user terminal 300 to display that the billing procedure has been completed.

[0067] [Billing Server] The billing server 400 is a server (computer) that executes billing procedures in cooperation with the evaluation device 100. The billing server 400 is configured to be able to communicate with the evaluation device 100 via a network, for example. The billing server 400 receives billing processing data including the billing amount and the user's consent from the billing processing unit 18 of the evaluation device 100. After that, when the billing server 400 completes the billing, it transmits billing completion information indicating that the billing has been completed to the billing processing unit 18 of the evaluation device 100. The billing server 400 may be a server managed by a business operator that performs a business related to settlement, for example.

[0068] [Learning process] The learning process executed by the evaluation device 100 will be described. Fig. 5 is a sequence diagram showing the learning process for generating a trained model N. In the learning process shown in Fig. 5, first, in step S10, the judgment data presenting unit 10 of the evaluation device 100 transmits a judgment data group JDG (each judgment data JD) to the judgment subject terminal 200. After that, the learning process proceeds to step S12.

[0069] In step S12, the judgment input unit 20 of the judgment subject terminal 200 outputs the judgment data group JDG received from the judgment data presenting unit 10 on an output device such as a display, and accepts input of the judgment result of the unique judgment subject S on the judgment data group JDG. As a result, the judgment input unit 20 generates a unique learning data group TDG (each unique learning data TD). In addition, when accepting the result of a judgment made on the judgment data group JDG from the unique judgment subject S, the judgment input unit 20 accepts attribute information indicating the attributes of the unique judgment subject S. As a result, the judgment input unit 20 acquires the attribute information of the unique judgment subject S that made a judgment on the judgment data group JDG by linking it to the unique learning data group TDG by the unique judgment subject S. Thereafter, the learning process proceeds to step S14.

[0070] In step S14, the judgment subject terminal 200 transmits the acquired unique learning data group TDG and the attribute information of the unique judgment subject S linked to the unique learning data group TDG to the evaluation device 100. After that, the learning process proceeds to step S16.

[0071] In step S16, the learning processing unit 11 of the evaluation device 100 receives the unique learning data group TDG from the judgment subject terminal 200, and generates a trained model N corresponding to the unique judgment subject S related to the unique learning data group TDG, based on the unique learning data group TDG received from the judgment subject terminal 200. After that, the learning process proceeds to step S18.

[0072] In step S18, the trained model storage unit 12 of the evaluation device 100 acquires the trained model N generated by the learning processing unit 11. The trained model storage unit 12 stores a plurality of trained models N corresponding to each unique judgment subject S in association with attribute information of the unique judgment subject S. With the above, the learning process executed by the evaluation device 100 is completed.

[0073] [Evaluation process] The evaluation process executed by the evaluation device 100 will be described. Fig. 6 is a sequence diagram showing the evaluation process for evaluating the evaluation target data ED. In the evaluation process shown in Fig. 6, first, in step S20, the evaluation target data input unit 30 of the user terminal 300 inputs the evaluation target data ED to the evaluation target data acquisition unit 14 of the evaluation device 100, and the target attribute input unit 31 of the user terminal 300 inputs the target attribute to the target attribute selection unit 15 of the evaluation device 100. After that, the evaluation process proceeds to step S22.

[0074] In step S22, the target attribute selection unit 15 of the evaluation device 100 selects a target attribute that is a target attribute. Here, the target attribute selection unit 15 selects the target attribute input by the target attribute input unit 31 of the user terminal 300. After that, the evaluation process proceeds to step S24.

[0075] In step S24, the evaluation execution unit 16 of the evaluation device 100 evaluates the evaluation target data ED acquired by the evaluation target data acquisition unit 14 using multiple trained models N stored in the trained model storage unit 12. More specifically, the evaluation execution unit 16 extracts a unique judgment subject S having a target attribute selected by the target attribute selection unit 15 from the multiple unique judgment subjects S based on the attribute information acquired by the attribute information acquisition unit 13, and evaluates the evaluation target data ED using the trained model N corresponding to the extracted unique judgment subject S. After that, the evaluation process proceeds to step S26.

[0076] In step S26, the evaluation result output unit 17 of the evaluation device 100 outputs information on the evaluation result of the evaluation target data ED output by the evaluation execution unit 16 (evaluation result information) to the user terminal 300. In addition, the evaluation result output unit 17 outputs attribute information (target attribute information) of each unique judgment subject S corresponding to each trained model N used in the evaluation of the evaluation target data ED by the evaluation execution unit 16 to the user terminal 300. After that, the evaluation process proceeds to step S28.

[0077] In step S28, the output processing unit 32 of the user terminal 300 acquires the evaluation result of the evaluation target data ED and attribute information (target attribute information) which is information on the attribute (target attribute) of the unique judgment subject S corresponding to the trained model N used in the evaluation of the evaluation target data ED, from the evaluation result output unit 17 of the evaluation device 100. Then, the output processing unit 32 displays the acquired evaluation result and target attribute information on the display 4 of the user terminal 300. With the above, the evaluation process executed by the evaluation device 100 is completed.

[0078] [Charging process] The charging process executed by the evaluation device 100 will be described. Fig. 7 is a sequence diagram showing the charging process for performing charging. In the charging process shown in Fig. 7, first, in step S30, the charging processing unit 18 of the evaluation device 100 outputs charging amount information indicating the amount of the charge to the user terminal 300. After that, the charging process proceeds to step S32.

[0079] In step S32, the billing instruction unit 33 of the user terminal 300 displays the billing amount information input from the billing processing unit 18 on the display 4 of the user terminal 300, so that the user can recognize the amount of the billing. Then, the billing instruction unit 33 confirms the user's intention to consent to the billing. For example, the billing instruction unit 33 displays a billing consent button on the display 4 of the touch panel of the user terminal 300, and confirms the user's intention to consent to the billing by touching the button. Then, the billing process proceeds to step S34.

[0080] In step S34, accounting instruction unit 33 of user terminal 300 outputs accounting instruction information that instructs accounting to accounting processing unit 18 of evaluation device 100. After that, the accounting process proceeds to step S36.

[0081] In step S36, when accounting processing unit 18 of evaluation device 100 accepts accounting instruction information from user terminal 300, it transmits data including the amount of the charge and the content related to the user's consent (accounting processing data) to accounting server 400. After that, the accounting process proceeds to step S38.

[0082] In step S38, billing server 400 completes billing based on the billing processing data received from billing processing unit 18. Then, billing server 400 transmits billing completion information indicating that billing has been completed to billing processing unit 18 of evaluation device 100. After that, the billing process proceeds to step S40.

[0083] In step S40, when the billing procedure between billing server 400 and billing processing unit 18 of evaluation device 100 is completed, billing processing unit 18 outputs a billing completion notice notifying user terminal 300 that billing has been completed. Here, billing processing unit 18 determines that the billing procedure has been completed by receiving billing completion information from billing server 400. After that, the billing process proceeds to step S42.

[0084] In step S42, when the accounting completion notification is input, the accounting instruction unit 33 of the user terminal 300 displays the fact that the accounting procedure is completed on the display 4 of the user terminal 300. With the above, the accounting process executed by the evaluation device 100 is completed.

[0085] [Evaluation Program] The following describes an evaluation program for causing a computer to function as the evaluation device 100. The evaluation program is a program for causing a computer to execute processing so as to evaluate the evaluation target data ED using a plurality of trained models N that are models trained with a unique learning data group TDG (a plurality of unique learning data TD) that is a result of each unique judgment entity S making a judgment on a judgment data group JDG (each judgment data JD) on a specific theme.

[0086] The evaluation program includes a main module, a judgment data presentation module, a learning processing module, a trained model storage module, an attribute information acquisition module, an evaluation target data acquisition module, a target attribute selection module, an evaluation execution module, an evaluation result output module, and a billing processing module. The main module is a part that controls the computer in an integrated manner. The functions realized by executing each of the judgment data presentation module, the learning processing module, the trained model storage module, the attribute information acquisition module, the evaluation target data acquisition module, the target attribute selection module, the evaluation execution module, the evaluation result output module, and the billing processing module are similar to the functions of the judgment data presentation unit 10, the learning processing unit 11, the trained model storage unit 12, the attribute information acquisition unit 13, the evaluation target data acquisition unit 14, the target attribute selection unit 15, the evaluation execution unit 16, the evaluation result output unit 17, and the billing processing unit 18 of the evaluation device 100. Note that the evaluation program may not include some of the modules described above other than the main module.

[0087] [Action and Effects] As described above, in general artificial intelligence, the subjects (unique judgment subjects S) that provide the original data for learning (judgment on the judgment data group JDG) are an unspecified number of subjects, whereas in the evaluation device 100, a large amount of data (unique learning data group TDG) is collected from each unique judgment subject S in a state where the unique judgment subject S is individually identified. As a result, a trained model N is constructed for each unique judgment subject S, so that it is possible to evaluate the evaluation target data ED by reflecting information for each unique judgment subject S. In addition, as a result, it is also possible to trace back to the unique judgment subject S from the evaluation result and confirm the attributes of the unique judgment subject S. As an example, such an evaluation device 100 can be efficiently realized by conducting a questionnaire survey of multiple unique judgment subjects S.

[0088] The evaluation device 100 comprises a trained model memory unit 12 that stores multiple trained models N corresponding to multiple unique judgment entities S, an evaluation target data acquisition unit 14 that acquires evaluation target data ED to be evaluated, and an evaluation execution unit 16 that uses the multiple trained models N stored in the trained model memory unit 12 to evaluate the evaluation target data ED acquired by the evaluation target data acquisition unit 14, and each of the multiple trained models N is a model trained from multiple unique training data TD that are the result of the unique judgment entity S corresponding to the trained model N making a judgment on each of multiple judgment data JD on a specific theme.

[0089] According to the evaluation device 100, a trained model N, which is a model trained with a plurality of unique learning data TD, which are the results of the unique judgment subject S making a judgment on each of a plurality of judgment data JD, is stored for each of the plurality of unique judgment subjects S. Then, the evaluation target data ED is evaluated using these plurality of trained models N. Therefore, the evaluation device 100 can make an evaluation that reflects information for each subject that provided the original data for learning.

[0090] In the evaluation device 100, each of the multiple determination data JD is data including multiple-choice questions. This makes it easier to statistically process the determination results of the unique determination subject S on each determination data JD.

[0091] In the evaluation device 100, each of the multiple trained models N is a model trained on 30 or more unique training data TD that are the results of a unique judgment entity S corresponding to the trained model N making a judgment on each of 30 or more judgment data JD. This can improve the accuracy of evaluation of the evaluation target data ED using the trained model N.

[0092] In the evaluation device 100, the number of options for a question is two. This makes it easier for the unique judgment subject S to make a judgment on the judgment data JD, compared to when the number of options is three or more.

[0093] In the evaluation device 100, the number of unique learning data groups TDG consisting of multiple unique learning data TD, which are the results of each of the multiple unique judgment entities S making a judgment on each of the multiple judgment data JD, is the same as the number of unique judgment entities S and is also the same as the number of trained models N. This makes it possible to make an evaluation that more accurately reflects information about each entity that provided the original data for learning.

[0094] In the evaluation device 100, each of the multiple unique learning data groups TDG is in one-to-one correspondence with any of the unique judgment subjects S, and is also in one-to-one correspondence with any of the trained models N. This makes it possible to make an evaluation that more accurately reflects information about each subject that provided the original data for learning.

[0095] The evaluation device 100 includes an attribute information acquisition unit 13 that acquires attribute information indicating the attributes of each of the multiple unique judgment subjects S, and a target attribute selection unit 15 that selects a target attribute that is a target attribute. The evaluation execution unit 16 extracts the unique judgment subjects S having the target attribute selected by the target attribute selection unit 15 from the multiple unique judgment subjects S based on the attribute information acquired by the attribute information acquisition unit 13, and evaluates the evaluation target data ED using a trained model N corresponding to the extracted unique judgment subject S. This makes it possible to retroactively acquire the attribute of the unique judgment subject S corresponding to the trained model N used in the evaluation from the evaluation result. In addition, the evaluation target data ED can be evaluated using the trained model N corresponding to the unique judgment subject S of a more suitable attribute, for example, according to the user's request or the contents of the evaluation target data ED.

[0096] In the evaluation device 100, the evaluation execution unit 16 performs individual evaluation of the evaluation target data ED using each of the multiple trained models N to obtain multiple unique evaluation results IR, and performs comprehensive evaluation of the evaluation target data ED based on the multiple acquired unique evaluation results IR to obtain a comprehensive evaluation result CR, thereby evaluating the evaluation target data ED. According to this, it is possible to evaluate the evaluation target data ED after weighting each trained model N corresponding to each unique judgment entity S according to, for example, the user's request or the contents of the evaluation target data ED.

[0097] The evaluation device 100 includes an attribute information acquisition unit 13 that acquires attribute information indicating the attributes of each of the multiple unique judgment subjects S, and an evaluation result output unit 17 that outputs to the outside the attribute information of each of the multiple unique judgment subjects S corresponding to each of the multiple trained models N used in the individual evaluation of the evaluation target data ED. This allows the user to know information about the subject that provided the original data for learning in the trained model N.

[0098] The evaluation device 100 includes an evaluation result output unit 17 that outputs to the outside a plurality of judgment data JD or a plurality of unique learning data TD corresponding to each of a plurality of trained models N used in the individual evaluation. This allows the user to learn information about the entity that provided the original data for learning in the trained model N, tracing back from the evaluation result of the evaluation target data ED to the judgment data JD or the unique learning data TD. As a result, it becomes possible to learn information about the grounds for the evaluation, and also to obtain suggestions for how to improve the evaluation target data ED.

[0099] The evaluation device 100 includes a billing processing unit 18 that executes processing of billing for evaluations performed by the evaluation executing unit 16, and the amount of billing is determined based on conditions related to the multiple trained models N used in the evaluation by the evaluation executing unit 16. This makes it possible to bill the user at a more appropriate amount.

[0100] In the evaluation device 100, the conditions include the number of multiple trained models N used by the evaluation execution unit 16. This makes it possible to charge the user an amount according to the number of multiple trained models N used by the evaluation execution unit 16.

[0101] The evaluation program causes a computer to execute processing to evaluate the evaluation target data ED using a plurality of trained models N, which are models trained with a plurality of unique learning data TD, which are the results of each of a plurality of unique judgment subjects S making a judgment on each of a plurality of judgment data JD on a specific theme. In other words, the evaluation program is a program for causing a computer to function as the evaluation device 100. This makes it possible to perform an evaluation that reflects information about each subject that provided the original data for learning.

[0102] [Second embodiment] Next, a second embodiment will be described. The evaluation device 100 according to the second embodiment is different from the evaluation device 100 according to the first embodiment in the function of the evaluation execution unit 16, but is otherwise common. For the sake of convenience, the same reference numerals are used for corresponding components in the first and second embodiments, even if the components have different functions.

[0103] The evaluation execution unit 16 of the evaluation device 100 according to the second embodiment evaluates the multiple evaluation target data ED by performing a first-stage evaluation of the multiple evaluation target data ED using each of the multiple trained models N to obtain multiple first-stage evaluation results, and by performing a second-stage evaluation of the multiple evaluation target data ED based on the multiple acquired first-stage evaluation results to obtain second-stage evaluation results, thereby evaluating the multiple evaluation target data ED.

[0104] This can be expressed as follows. That is, the evaluation execution unit 16 of the evaluation device 100 according to the second embodiment evaluates (i.e., the above-mentioned multiple evaluation target data ED) including multiple types of elements using each of multiple trained models N (here, "multiple trained models N" means that there are multiple trained models N corresponding to each of multiple unique decision subjects S) that have learned at least these multiple types of elements (trained as input data) to obtain multiple first-stage evaluation results (i.e., for each trained model N), and obtains a second-stage evaluation result by performing a second-stage evaluation of the evaluation target data ED including the above-mentioned multiple types of elements based on the acquired first-stage evaluation results, thereby evaluating the evaluation target data ED including the above-mentioned multiple types of elements. Here, the "first-stage evaluation" is an evaluation performed on the evaluation target data ED including multiple types of elements by taking into account each element, and the "first-stage evaluation result" is an evaluation result by the first-stage evaluation. In addition, the "second-stage evaluation" is an evaluation that combines multiple first-stage evaluation results, which are the evaluation results of multiple first-stage evaluations performed using each of multiple trained models N, and the "second-stage evaluation result" is the evaluation result from the second-stage evaluation.

[0105] As an example, when the evaluation target by the trained model N is a website, the multiple types of elements to be included in the evaluation target data ED may be a "catchphrase" and a "layout", etc. The trained model N is trained using at least these multiple types of elements as input data. Therefore, the evaluation target data ED including these multiple types of elements is input to each trained model N corresponding to each of the multiple unique decision-making entities S, and an evaluation (i.e., a first-stage evaluation) incorporating these multiple types of elements is performed. After that, the multiple first-stage evaluation results acquired by each trained model N are combined (for example, statistically processed) and an evaluation (i.e., a second-stage evaluation) is performed, and a second-stage evaluation result is acquired. Then, the evaluation target data ED is finally evaluated based on the acquired second-stage evaluation result.

[0106] As described above, according to the evaluation device 100 of the second embodiment, for example, a composite evaluation (i.e., a first-stage evaluation) is performed on the evaluation target data ED including a plurality of types of elements by each of a plurality of trained models N, taking into account a plurality of types of elements. Then, an evaluation (i.e., a second-stage evaluation) is performed on the evaluation target data ED by combining a plurality of evaluation results (i.e., a plurality of first-stage evaluation results) taking into account these plurality of types of elements. Therefore, since the evaluation of the entire evaluation target data ED including a plurality of types of elements is performed using an evaluation taking into account the elements of the evaluation target data ED, it is possible to improve the accuracy of the evaluation.

[0107] [Variations] The above-described embodiment can be implemented in various forms that are modified or improved based on the knowledge of those skilled in the art.

[0108] For example, in the above-described embodiment, the evaluation device 100 is configured as a single computer (server). However, the evaluation device 100 may be configured with its functions distributed across a plurality of computers (servers). [Explanation of symbols]

[0109] 10. Data presentation section for judgment 11 Learning processing unit 12 Trained model memory 13 Attribute information acquisition section 14 Evaluation target data acquisition section 15 Target attribute selection section 16 Evaluation Execution Department 17 Evaluation result output section 18. Billing Processing Unit 100 Evaluation Device CR Overall Evaluation Result ED Evaluation Data IR specific evaluation results JD Judgment Data JDG Judgment Data Set N trained models S specific decision subject TD specific training data TDG specific training data set

Claims

1. an evaluation target data acquisition unit that acquires evaluation target data on a specific theme inputted into a user's terminal; a judgment data presentation unit that presents a plurality of judgment data items on the specific theme, different from the evaluation target data, to terminals of a plurality of unique judgment subjects that have at least one attribute in common with the user and are different from the user; a learning processing unit that acquires, from the terminals of the multiple unique judgment subjects, multiple unique learning data that are the results of the multiple unique judgment subjects making judgments on each of the multiple judgment data, and generates multiple trained models corresponding to each of the multiple unique judgment subjects based on the multiple acquired unique learning data; A trained model storage unit that stores a plurality of the trained models; An evaluation execution unit that evaluates the evaluation target data acquired by the evaluation target data acquisition unit using a plurality of the trained models stored in the trained model storage unit, an evaluation device, wherein each of the plurality of judgment data is a multiple-choice question including a question and a plurality of options to be selected by the unique judgment entity as a judgment on the question;

2. 2. The evaluation device according to claim 1, wherein each of the plurality of trained models is a model trained on 30 or more pieces of unique learning data that are the results of the unique judgment entity corresponding to the trained model making a judgment on each of 30 or more pieces of judgment data.

3. The evaluation device according to claim 1 , wherein the number of options for the question is two.

4. The evaluation device according to any one of claims 1 to 3, wherein the number of unique learning data groups consisting of a plurality of the unique learning data, which are the results of each of the plurality of unique judgment entities making a judgment on each of the plurality of judgment data, is the same as the number of the unique judgment entities and is the same as the number of the trained models.

5. The evaluation device according to claim 4 , wherein each of the plurality of unique learning data groups is in one-to-one correspondence with one of the unique judgment subjects and in one-to-one correspondence with one of the trained models.

6. an attribute information acquisition unit that acquires attribute information indicating at least one attribute of each of the plurality of unique decision subjects, the at least one attribute including at least one attribute common to the user; a target attribute selection unit that selects a target attribute that is a target attribute from the at least one attribute indicated by the attribute information, The evaluation execution unit extracts, from among the multiple unique judgment entities, a unique judgment entity having the target attribute selected by the target attribute selection unit based on the attribute information acquired by the attribute information acquisition unit, and evaluates the evaluation target data using the trained model corresponding to the extracted unique judgment entity. The evaluation device described in any one of claims 1 to 5.

7. The evaluation device according to any one of claims 1 to 6, wherein the evaluation execution unit evaluates the plurality of evaluation target data by performing a first-stage evaluation of the plurality of evaluation target data using each of the plurality of trained models to obtain a plurality of first-stage evaluation results, and by performing a second-stage evaluation that combines the obtained plurality of first-stage evaluation results to obtain a second-stage evaluation result.

8. The evaluation execution unit performs an individual evaluation of the evaluation target data using each of the plurality of trained models to obtain a plurality of unique evaluation results, and performs a comprehensive evaluation of the evaluation target data based on the obtained plurality of unique evaluation results to obtain a comprehensive evaluation result, thereby evaluating the evaluation target data. The evaluation device according to any one of claims 1 to 7.

9. an attribute information acquisition unit that acquires attribute information indicating at least one attribute of each of the plurality of unique decision subjects, the at least one attribute including at least one attribute common to the user; 9. The evaluation device according to claim 8, further comprising: an evaluation result output unit that outputs to an outside the attribute information of each of the plurality of unique judgment subjects corresponding to each of the plurality of trained models used in the individual evaluation.

10. 9. The evaluation device according to claim 8, further comprising an evaluation result output unit that outputs to an outside the plurality of judgment data or the plurality of unique learning data corresponding to each of the plurality of trained models used in the individual evaluation.

11. a charging processing unit that performs a charging process related to the evaluation performed by the evaluation execution unit, The evaluation device according to any one of claims 1 to 10, wherein the amount of the charge is determined based on conditions related to the plurality of trained models used for evaluation by the evaluation execution unit.

12. The evaluation device according to claim 11 , wherein the conditions include a number of the trained models used by the evaluation execution unit.

13. An evaluation program that causes a computer to execute a process to evaluate evaluation target data on a specific theme, The process comprises: an evaluation target data acquisition step of acquiring the evaluation target data inputted at a user's terminal; a judgment data presentation step of presenting a plurality of judgment data on the specific theme, which is different from the evaluation target data, to terminals of a plurality of unique judgment subjects that have at least one attribute in common with the user and are different from the user; a learning process step of acquiring multiple unique learning data, which are the results of the multiple unique judgment entities making judgments on each of the multiple judgment data, from the terminals of the multiple unique judgment entities, and generating multiple trained models corresponding to each of the multiple unique judgment entities based on the acquired multiple unique learning data; A trained model storage step of storing the trained models in a storage unit of the computer; an evaluation execution step of evaluating the evaluation target data acquired in the evaluation target data acquisition step by using a plurality of the trained models stored in the storage unit; Including, an evaluation program, wherein each of the plurality of judgment data is a multiple-choice question including a question and a plurality of options to be selected by the unique judgment entity as a judgment in response to the question;

Images