Evaluation system and evaluation program

Abstract

Provided are an evaluation system and an evaluation program capable of, with little load, using a plurality of evaluation criteria to accurately evaluate a business plan document.　This evaluation system for using a plurality of evaluation criteria to evaluate a business plan document comprises: a plan document input unit 11 that inputs the business plan document; a criteria input unit 12 that inputs the evaluation criteria; an evaluation level calculation unit 20 that uses the inputted business plan document and evaluation criteria to calculate the evaluation level of each evaluation criterion for the business plan document; and an evaluation unit 30 that performs a final evaluation of the business plan document on the basis of the evaluation levels.

Description

Evaluation System and Evaluation Program

[0001] The present invention relates to an evaluation system for evaluating a business plan using a plurality of evaluation criteria, and an evaluation program for causing a computer to perform such evaluation.

[0002] The business plan formulation support system according to Patent Document 1 stores a plurality of pieces of regional information and case information of businesses formulated for each region, acquires an index for the problem to be solved by the business plan, extracts case information having a common index from the case information as recommended case information, generates and outputs information based on a business evaluation item indicating a business result from among this recommended case information. Further, the business plan formulation support system generates an evaluation model learned by learning data having one business evaluation item as an objective variable and other business evaluation items as explanatory variables based on the recommended case information, and outputs the value of the objective variable obtained by inputting the values of the business evaluation items of the business plan as explanatory variables into the evaluation model.

[0003] The language transmission ability evaluation system according to Patent Document 2 includes an evaluation target acquisition unit that receives user input including a question and an evaluation target, an evaluation criterion acquisition unit that acquires evaluation criteria for a language ability evaluation test, a prompt providing unit that provides a prompt including the user input and instructions and explanations for a generation AI to the generation AI, an answer acquisition unit that acquires an answer to the prompt from the generation AI, and an evaluation display unit that displays an evaluation of the evaluation target included in the answer. The instructions constituting the prompt include a role assignment instruction for assigning the role of an evaluator of the language ability evaluation test to the generation AI, an evaluation creation instruction for causing the evaluation target to be evaluated based on the evaluation criteria, and a confirmation instruction for confirming that the evaluation of the evaluation creation instruction is based on the evaluation criteria. Further, the explanations constituting the prompt include an explanation of the evaluation criteria and an explanation of an evaluation method based on the evaluation criteria.

[0004] Japanese Patent Application Laid-Open No. 2024-011875 Patent No. 7521860

[0005] In the business plan formulation support system described in Patent Document 1, a learning model is required to be constructed using training data in which one of the business evaluation items is the dependent variable and the other business evaluation items are the independent variables. However, if there are many business evaluation items, creating the training data becomes a significant burden.

[0006] In the language communication ability evaluation system described in Patent Document 2, evaluation information for the evaluation target can be obtained from the generating AI by providing prompts, thus reducing the burden of preparation work for evaluation. In particular, in language communication ability evaluations such as English proficiency tests, the evaluation criteria are limited to language communication ability and are well-known criteria, making them relatively simple. However, if the evaluation target is a business plan, the items that make up the business plan and the items of the evaluation criteria are complex, making it difficult to directly apply the evaluation technology shown in Patent Document 2 to the evaluation of the business plan.

[0007] In view of the circumstances described above, the object of the present invention is to provide an evaluation system that can accurately evaluate business plans with minimal burden using multiple evaluation criteria.

[0008] The evaluation system according to the present invention is a system for evaluating a business plan using multiple evaluation criteria, and comprises a plan input unit for inputting the business plan, a criteria input unit for inputting the evaluation criteria, an evaluation level calculation unit for calculating an evaluation level for each evaluation criterion for the business plan using the input business plan and evaluation criteria, and an evaluation unit for performing a final evaluation of the business plan based on the evaluation levels.

[0009] This structure provides multiple evaluation criteria for assessing business plans. Each business plan is evaluated according to these criteria, and a corresponding evaluation level is calculated. Even if a business plan contains a lot of information, the evaluation workload is reduced if it is evaluated using only one criterion. Furthermore, since the final evaluation of a business plan is determined based on all the evaluation levels calculated for each criterion, the business plan is checked from all angles, resulting in an accurate final evaluation.

[0010] To check whether a business plan meets the evaluation criteria, it is preferable to check the similarity between the sentences contained in the business plan and the sentences contained in the evaluation criteria. The similarity between sentences can be calculated by vector operations such as the dot product operation if the sentences are represented as vectors. Using techniques such as "Word Embedding" to vectorize the sentences is also advantageous for subsequent evaluation processing (natural language processing). For this reason, the present invention proposes that the evaluation level calculation unit generates plan vector data by vectorizing the sentences that constitute the business plan, generates reference vector data by vectorizing the sentences that constitute the evaluation criteria, and calculates the evaluation level based on the similarity between the business plan and each of the evaluation criteria calculated using the plan vector data and the reference vector data.

[0011] Using plan vector data and reference vector data, a classification method such as the K-means method can be used to calculate the similarity (degree of satisfaction with evaluation criteria) between the business plan and each evaluation criterion. This is advantageous because it allows for quantitative analysis of the calculated clusters (corresponding to evaluation levels), such as the Euclidean distance from the representative point of the cluster. Therefore, in this invention, it is proposed that the evaluation level is represented by a plurality of clusters divided in a vector space containing the plan vector data and the reference vector data, and that the evaluation level is calculated through calculation of the distribution state of the clusters. The evaluation level of the business plan can be calculated from the distribution state of each of the clusters. For example, the vector data can be classified into a plurality of clusters (evaluation levels) using the K-means method, and the vector data can be classified into each cluster (each evaluation level).

[0012] As described above, calculating the evaluation level for a business plan using plan vector data and reference vector data can be done relatively easily and with high accuracy by using various generative AIs, provided that appropriate prompts are created. For this reason, the present invention proposes that the evaluation level calculation unit is configured using generative AI, and that the business plan and the evaluation criteria are provided to the generative AI as prompts.

[0013] Recently, one type of business plan that has been the subject of much debate due to the difficulty of evaluation is one that generates carbon credits. Carbon credits are a system that allows companies to buy and sell greenhouse gas emission reductions, primarily between companies. Companies quantify the amount of greenhouse gas reductions or absorptions generated by their environmental activities, and can trade emission rights certified as carbon credits with other companies. For this reason, business plans related to carbon credits require evaluation from multiple perspectives, and the number of evaluation criteria items is large, making the evaluation crucial. Furthermore, the evaluation criteria include those established in each country as well as the unique criteria and know-how of each company. Therefore, in this invention, the business plan is a business plan for acquiring carbon credits, and the evaluation criteria include general evaluation criteria created in accordance with carbon credit certification standards and unique evaluation criteria created independently. This allows business plans that generate carbon credits to be accurately evaluated with minimal burden.

[0014] Since carbon credits are traded between companies or brokers, the trading value of the carbon credits produced is important in business plans for generating carbon credits. For this reason, a high trading value of the carbon credits produced may also be included as an evaluation criterion. Accordingly, this invention proposes that the final evaluation of a business plan for acquiring carbon credits may be influenced by the trading value of the carbon credits.

[0015] The present invention also includes the scope of rights for an evaluation program that causes a computer to perform a process of evaluating a business plan using multiple evaluation criteria. Such an evaluation program causes a computer to perform a plan input process in which the business plan is input, a criteria input process in which the evaluation criteria are input, an evaluation level calculation process that calculates an evaluation level for each evaluation criterion for the business plan using the input business plan and evaluation criteria, and an evaluation process that performs a final evaluation of the business plan based on the evaluation levels. Through each process performed during its execution, the various effects and advantages described above for the evaluation system can be obtained.

[0016] Other features, functions, and effects of the present invention will be revealed by the following description of the invention with reference to the drawings.

[0017] This is a schematic diagram showing the general structure of the evaluation system. This is a flowchart showing the processing flow when a business plan is quantitatively evaluated. This is a document diagram showing an example of a prompt. This is an explanatory diagram explaining the classification of evaluations using the K-means method. This is a list of evaluation levels for each evaluation criterion for multiple business plans. This is a list of evaluation levels for multiple business plans.

[0018] First, the schematic configuration of the evaluation system according to the present invention will be explained using Figure 1. This evaluation system evaluates a business plan using multiple evaluation criteria. The system comprises a business plan input unit 11, a criteria input unit 12, an evaluation level calculation unit 20, and an evaluation unit 30.

[0019] The plan input unit 11 converts the input business plan into data that can be processed by the evaluation level calculation unit 20 and provides it to the evaluation level calculation unit 20. The criteria input unit 12 converts the evaluation criteria used when evaluating the business plan into data that can be processed by the evaluation level calculation unit 20 and provides it to the evaluation level calculation unit 20. The evaluation criteria may consist of several to several hundred items, but these items may be compiled into one or more documents.

[0020] The evaluation level calculation unit 20 is a core component of the evaluation system and is substantially constructed using a computer unit. It calculates the evaluation level for each evaluation criterion for the business plan using the given business plan and evaluation criteria. For this reason, the evaluation level calculation unit 20 includes, as functional units, an instruction generation unit 21, a vector representation module 22, an arithmetic unit 23, and so on. At least some of the processing in each component constituting the evaluation system is substantially implemented by a computer through the execution of computer programs.

[0021] The command generation unit 21 (prompt generation unit 21) functions as an interface that receives instructions from the user and generates a command script for the calculation unit 23 of this evaluation system. The command script for the calculation unit 23 generated by the command generation unit 21 includes the user's instructions, the contents of the business plan, and the contents of the evaluation criteria. Examples of the user's instructions include, "The input document is a report on the carbon credit project," "For each criterion, find a description that serves as evidence to the question of whether this project conforms to the criterion," and "Output the obtained evidence while strictly adhering to the output format."

[0022] The vectorization module 22 generates plan vector data by representing the text (including words and phrases) that make up the business plan included in the instruction script as vectors, and similarly generates reference vector data by representing the text that makes up the evaluation criteria included in the instruction script as vectors. For vectorizing the text, the Word Embedding technique, which is used in BERT (Bidirectional Encoder Representations from Transformers), is used, for example. The generated plan vector data and reference vector data are provided to the calculation unit 23 along with the user's instructions.

[0023] The calculation unit 23 uses the plan vector data and the reference vector data, and, referring to user instructions, calculates the similarity score indicating the degree to which each evaluation criterion in the business plan is satisfied, thereby calculating the evaluation level of the input business plan. The similarity score can be calculated using techniques such as vector operations (e.g., dot product operations) between each sentence in the plan vector data and each sentence in the reference vector data to determine the similarity between them. An evaluation level is calculated for each of the multiple criterion items included in the evaluation criteria. Here, the evaluation level can be considered as a collection of evaluation levels for each criterion item.

[0024] The evaluation unit 30 performs a final evaluation of the business plan being evaluated based on the evaluation level, which is the calculation result of the calculation unit 23, i.e., the output of the evaluation level calculation unit 20, and outputs the final evaluation of the business plan. The evaluation unit 30 may be incorporated into the evaluation level calculation unit 20 as one of its functional units.

[0025] The vector representation module 22 and the evaluation level calculation unit 20 may be integrated as a generation AI 25. In that case, the instruction generation unit 21 functions as a prompt generation unit 21. Furthermore, the functions of the evaluation unit 30 can also be incorporated into this generation AI 25. Alternatively, the evaluation unit 30 may be configured as an independent machine learning model, independent of the generation AI 25.

[0026] Next, one embodiment of the evaluation system of the present invention will be described. In this embodiment, the evaluation system evaluates a business plan for acquiring carbon credits based on carbon credit evaluation criteria. The evaluation level calculation unit 20 of this evaluation system includes a generating AI 25 that outputs an evaluation level when given a prompt generated by a prompt generation unit 21. The carbon credit evaluation criteria include general evaluation criteria created in accordance with carbon credit certification criteria and proprietary evaluation criteria created independently.

[0027] Carbon credits are emission allowances that companies and individuals can acquire by funding projects that reduce greenhouse gas emissions, and the value of these allowances is equivalent to the amount of greenhouse gas emissions reduced. Such greenhouse gas emission projects (hereinafter simply referred to as "projects") require significant investment, so it is important to carefully examine the project proposal in advance to ensure that the carbon credits generated by the project have appropriate value. Examples of project proposals include projects related to reducing greenhouse gas emissions from rice paddies, projects related to afforestation and deforestation control, and projects related to biochar, and such project proposals constitute the business plan in this invention. In the examples shown in Figures 3 and 6, a project proposal related to biochar is used as the business plan, but of course, the business plan in this invention is not limited to projects related to biochar.

[0028] Figure 2 shows the process flow when a business plan and a forestry business plan, both project proposals related to biochar, are quantitatively evaluated by the evaluation system. This evaluation system also allows for the sequential evaluation of various business plans using the same evaluation criteria. The following explanation will focus on the business plan related to biochar.

[0029] A business plan for the biochar project to be evaluated is created by the user. Simultaneously, the user creates carbon credit evaluation criteria (hereinafter simply referred to as "evaluation criteria") for evaluating this business plan. The created business plan and evaluation criteria are converted into text data and provided to the evaluation system.

[0030] The evaluation criteria consist of more than 100 items, some of which are as follows: (1) The raw material must be pure biological waste biomass and must not be cultivated for a specific purpose. (2) The raw material must be left to rot or burn for purposes other than energy production. (3) Biochar is expressed in dry tonnage. Care must be taken to avoid moisture, as moisture would overestimate the amount of carbon sequestration. (4) The biomass used must not contain paint residue, solvents, or other potentially toxic impurities. (5) The raw material must not be imported from other countries.

[0031] The evaluation system can generate prompts using a given business plan and evaluation criteria, but prompts may also be created by the user using the user interface of the prompt generation unit 21 and provided to the evaluation system. In this embodiment, the generated AI 25 is constructed using an LLM (Large-Scale Language Model), so prompts suitable for the specifications of this LLM are generated. An example of such a prompt is shown in Figure 3. In the example in Figure 3, the prompt generation unit 21 generates prompts based on instructions (user instructions), a business plan, and evaluation criteria provided by the user via the internet or the like. Here, the prompts are written in English to improve the output accuracy of the generated AI 25. Of course, the prompts may also be written in Japanese.

[0032] If the amount of data in the input business plan is large, the prompt will contain the file name of the document designated to be read from an appropriate memory location. Also, if there are many items in the input evaluation criteria, prompts containing a predetermined number of evaluation criterion items will be sequentially given to the generating AI 25, and the multiple output results output by the generating AI 25 will be integrated.

[0033] The extent to which a business plan meets the evaluation criteria is determined based on the degree of similarity between the business plan and each evaluation criterion item (the degree to which the business plan satisfies the evaluation criteria). The evaluation level, which represents the degree to which the business plan satisfies each evaluation criterion as a result of this determination, is set to the following six levels: "Yes", "Likely Yes", "Likely No", "No", "Irrelevant", and "Not Mentioned".

[0034] This evaluation system uses the similarity (distance) of each data point in the vector space of the vector data of the text constituting the business plan (plan vector data) and the vector data of the text constituting the evaluation criteria (evaluation criteria vector data) to make judgments based on similarity. In other words, each data point is classified into clusters based on the evaluation levels of "Yes", "Likely Yes", "Likely No", "No", "Irrelevant", and "Not Mentioned". That is, the classification of each vector data point to a specific cluster (specific evaluation level) is calculated from the distribution state of the vector data group for each cluster in the vector space. The K-means method is used for this classification.

[0035] In this embodiment, the K-means classification method, as shown in Figure 4, involves setting up six classification areas (corresponding to evaluation levels) corresponding to "Yes", "Likely Yes", "Likely No", "No", "Irrelevant", and "Not Mentioned" in a vector space where data sets consisting of plan vector data and reference vector data are scattered. The data sets are then classified into each of these classification areas, and the distance (Euclidean distance: L2 distance) between the centroid of the data sets contained in each classification area and the centroid of each classification area is calculated. This distance provides quantitative support for the calculation of the evaluation level. To explain in more detail, for example, the classification is performed by the following procedure. (1) The description of each piece of evidence is converted into a vector representation to generate a set of vector data. (2) The vector data is clustered into each classification region. (3) The centroid vector of the cluster in each classification region is calculated. (4) For each piece of evidence, the classification region of the cluster with the lowest Euclidean distance among the centroid vectors calculated in (3) is used as the classification region by the K-means method.

[0036] Therefore, the prompt will include output instructions such as six evaluation levels and a description of the evidence that underlies those evaluations.

[0037] The evaluation unit 30 evaluates the value (quality) of the business plan being evaluated based on the evaluation level calculated for each of the more than 100 evaluation criteria, and generates evaluation data (evaluation report) that includes the carbon credit value of the business plan.

[0038] Figure 5 is a reference example of a table output showing six evaluation levels for multiple evaluation criteria for several business plans. In this reference example, the data set subject to the K-means method is vector data obtained by vectorizing the text (evidence) extracted as the basis for evaluation calculation, output by the generating AI. Examples of text extracted as the basis for evaluation calculation are as follows: (1) A certain company's charcoal production is batch-type. Firewood is packed into the kiln, the combustion chamber is sealed, and thermal decomposition is carried out in each kiln. During the cooling stage, heat is dissipated from the kiln walls and top, and it cools in that manner. After cooling, the combustion chamber is opened, and the carbonized material is processed for transport. (2) There is no mention of the moisture content of the biochar to prevent dust generation and explosion during transport. (3) This document does not contain details about the moisture content of the biochar to prevent dust generation and explosion during transport. (4) Moisture content of individual large bags was confirmed to be 20-30%. (5) This document does not contain information on the transportation of biochar over distances exceeding 200 km and the associated greenhouse gas emissions.

[0039] In the table in Figure 5, each item in the vertical column indicates the document number that defines the business plan, and each item in the horizontal row indicates the evaluation criterion item number that defines the evaluation criterion item. For example, each cell in the second column displays the evaluation level for each evaluation criterion item for business plan document number 2, and the cell in row 2, column 2 (2,2) displays "Likely Yes" as the evaluation level for evaluation criterion item 2 for business plan document number 2.

[0040] The evaluation unit 30 can also estimate the trading value of carbon credits for each business plan and generate a comprehensive evaluation report showing their value ranking, based on evaluation level data as shown in Figure 5.

[0041] In the table output shown as an example in Figure 6, the vertical columns are the same as in Figure 5, the first cell in each row (Document iD) shows the document number that defines the business plan, the second cell in each row (labeled class) shows the evaluation level of the business plan, the third cell in each row (evidence) shows the text (evidence) extracted as the basis for calculating the evaluation, the fourth cell in each row (matched) shows whether the evaluation by the generated AI and the evaluation by the K-means method are in agreement (TRUE) or not in agreement (FALSE), the fifth cell in each row (K-means classification) shows the evaluation level by the K-means method, and the sixth cell in each row (L2 distance) shows the distance value calculated by the K-means method. The fifth cell in the horizontal row shows the evaluation level that ranked first using the K-means method, and the sixth cell in the horizontal row shows the distance value that served as the basis for this first-place ranking. The cells from the seventh cell onward in the horizontal row sequentially show the evaluation levels and distance values ​​for second place and below using the K-means method. The final evaluation of the business plan is the evaluation level that ranked first, and in Figure 6, the final evaluation of the business plan shown in document number 1 is "Yes". At that time, the evaluation levels and distance values ​​for second place and below are also shown, allowing for a quantitative analysis of the final evaluation.

[0042] [Alternative Embodiments] (1) In the embodiments described above, the evaluation system was shown to be configured as an integrated system, but these components may be distributed via a network, and the evaluation system may be configured as a distributed system. For example, the prompt generation unit 21 of the evaluation level calculation unit 20 may be built on the user terminal, the generation AI 25 of the evaluation level calculation unit 20 and the evaluation unit 30 may be built on a remote cloud computing service, and the final evaluation result may be sent to the user terminal. Alternatively, the generation AI 25 may be built on a separate server or user terminal.

[0043] (2) The evaluation unit 30 can be configured integrally with the generation AI 25, but it may be constructed as a machine learning model that takes various data outputs from the generation AI 25 (for example, list data as shown in FIGS. 5 and 6) as input data and outputs the final result indicating the value ranking of the business plan.

[0044] (3) In the above-described embodiment, as an embodiment of the business plan, a business plan mainly related to a biochar project was taken up, but the present invention is not limited to this, and the evaluation system of the present invention can be applied to various business plans such as a plan for an energy-saving related business and a plan for a food self-sufficiency business with the same advantages.

[0045] In addition, the configurations disclosed in the above embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with the configurations disclosed in other embodiments as long as there is no contradiction, and the embodiments disclosed in this specification are examples, and the embodiments of the present invention are not limited to this, and can be appropriately modified within the scope not departing from the object of the present invention.

[0046] The present invention can be applied to an evaluation system and an evaluation program for evaluating a business plan using a plurality of evaluation criteria.

[0047] 2: Evaluation criterion item 11: Plan input unit 12: Criterion input unit 20: Evaluation level calculation unit 21: Command generation unit (prompt generation unit) 22: Vector representation module 23: Calculation unit 25: Generation AI 30: Evaluation unit

Claims

1. An evaluation system for evaluating a business plan using multiple evaluation criteria, comprising: a plan input unit for inputting the business plan; a criteria input unit for inputting the evaluation criteria; an evaluation level calculation unit for calculating an evaluation level for each evaluation criterion for the business plan using the input business plan and evaluation criteria; and an evaluation unit for performing a final evaluation of the business plan based on the evaluation levels.

2. The evaluation system according to claim 1, wherein the evaluation level calculation unit generates plan vector data by representing the text constituting the business plan as vectors, generates reference vector data by representing the text constituting the evaluation criteria as vectors, and calculates the evaluation level based on the degree of similarity between the business plan and each of the evaluation criteria calculated using the plan vector data and the reference vector data.

3. The evaluation system according to claim 2, wherein the evaluation level is represented by a plurality of clusters separated in a vector space containing the plan vector data and the reference vector data, and the evaluation level is calculated through calculation of the distribution state of the clusters.

4. The evaluation system according to any one of claims 1 to 3, wherein the evaluation level calculation unit is configured using a generating AI, and the business plan and the evaluation criteria are provided to the generating AI as prompts.

5. The evaluation system according to claim 4, wherein the business plan is a business plan for acquiring carbon credits, and the evaluation criteria include general evaluation criteria created in accordance with carbon credit certification standards and proprietary evaluation criteria created independently.

6. The evaluation system according to claim 5, wherein the final evaluation of the business plan for obtaining the carbon credits is influenced by the trading value of the carbon credits.

7. An evaluation program that causes a computer to perform a process of evaluating a business plan using multiple evaluation criteria, the program comprising: a plan input process for inputting the business plan; a criteria input process for inputting the evaluation criteria; an evaluation level calculation process for calculating an evaluation level for each evaluation criterion for the business plan using the input business plan and evaluation criteria; and an evaluation process for performing a final evaluation of the business plan based on the evaluation levels.

Images