Evaluation system and evaluation program

Abstract

We provide an evaluation system and program that allows for the accurate evaluation of business plans with minimal effort using multiple evaluation criteria. [Solution] An evaluation system for evaluating a business plan using multiple evaluation criteria comprises a plan input unit 11 for inputting the business plan, a criteria input unit 12 for inputting the evaluation criteria, an evaluation level calculation unit 20 that calculates an evaluation level for each evaluation criterion for the business plan using the input business plan and evaluation criteria, and an evaluation unit 30 that performs a final evaluation of the business plan based on the evaluation levels.

Description

Technical Field

[0001] The present invention relates to an evaluation system for evaluating a business plan using a plurality of evaluation criteria, and an evaluation program for causing a computer to execute such evaluation.

Background Art

[0002] The business plan formulation support system according to Patent Document 1 stores a plurality of pieces of regional information and case information of businesses formulated for each region, obtains an index for a problem to be solved by the business plan, extracts case information having a common index from the case information as recommended case information, generates and outputs information based on a business evaluation item indicating a business result from among the recommended case information. Further, the business plan formulation support system generates an evaluation model learned by learning data having one business evaluation item as an objective variable and other business evaluation items as explanatory variables based on the recommended case information, and outputs the value of the objective variable obtained by inputting the values of the business evaluation items of the business plan as explanatory variables into the evaluation model.

[0003] The language transmission ability evaluation system according to Patent Document 2 includes an evaluation target acquisition unit that receives user input including a question and an evaluation target, an evaluation criterion acquisition unit that acquires evaluation criteria for a language ability evaluation test, a prompt providing unit that provides a prompt including the user input and instructions and explanations for a generation AI to the generation AI, an answer acquisition unit that acquires an answer to the prompt from the generation AI, and an evaluation display unit that displays an evaluation of the evaluation target included in the answer. The instructions constituting the prompt include a role assignment instruction for assigning a role as an evaluator of a language ability evaluation test to the generation AI, an evaluation creation instruction for causing the evaluation target to be evaluated based on the evaluation criteria, and a confirmation instruction for confirming that the evaluation of the evaluation creation instruction is based on the evaluation criteria. Further, the explanations constituting the prompt include an explanation of the evaluation criteria and an explanation of an evaluation method based on the evaluation criteria.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] In the business plan development support system described in Patent Document 1, a learning model needs to be constructed that learns from training data in which one of the business evaluation items is the dependent variable and the other business evaluation items are the independent variables. However, if there are many business evaluation items, creating the training data becomes a significant burden.

[0006] In the language communication ability evaluation system described in Patent Document 2, evaluation information for the evaluation target can be obtained from the generating AI by providing prompts, thus reducing the burden of preparation work for evaluation. In particular, in language communication ability evaluations such as English proficiency tests, the evaluation criteria are limited to language communication ability and are well-known, making them relatively simple. However, if the evaluation target is a business plan, the items that make up the business plan and the items of the evaluation criteria are complex, making it difficult to directly apply the evaluation technology shown in Patent Document 2 to the evaluation of the business plan.

[0007] In view of the circumstances described above, the object of the present invention is to provide an evaluation system that can accurately evaluate business plans with minimal burden using multiple evaluation criteria. [Means for solving the problem]

[0008] The evaluation system according to the present invention is a system for evaluating a business plan using multiple evaluation criteria, and comprises a plan input unit for inputting the business plan, a criteria input unit for inputting the evaluation criteria, an evaluation level calculation unit for calculating an evaluation level for each evaluation criterion for the business plan using the input business plan and evaluation criteria, and an evaluation unit for performing a final evaluation of the business plan based on the evaluation levels.

[0009] This structure provides multiple evaluation criteria for assessing business plans. Each business plan is evaluated according to these criteria, and a corresponding evaluation level is calculated. Even if a business plan contains a lot of information, the evaluation workload is reduced if it is evaluated using only one criterion. Furthermore, since the final evaluation of a business plan is determined based on all the evaluation levels calculated for each criterion, the business plan is checked from all angles, resulting in an accurate final evaluation.

[0010] To check whether a business plan meets the evaluation criteria, it is preferable to check the similarity between the sentences contained in the business plan and the sentences contained in the evaluation criteria. The similarity between sentences can be calculated by vector operations such as the dot product operation if the sentences are represented as vectors. Using techniques such as "Word Embedding" for vectorizing sentences is also advantageous for subsequent evaluation processing (natural language processing). For this reason, the present invention proposes that the evaluation level calculation unit generates plan vector data by representing the sentences constituting the business plan as vectors, generates reference vector data by representing the sentences constituting the evaluation criteria as vectors, and calculates the evaluation level based on the similarity between the business plan and each of the evaluation criteria calculated using the plan vector data and the reference vector data.

[0011] Using plan vector data and reference vector data, a classification method such as the K-means method can be used to calculate the similarity (degree of satisfaction with evaluation criteria) between the business plan and each evaluation criterion. This is advantageous because it allows for quantitative analysis of the calculated clusters (corresponding to evaluation levels), such as the Euclidean distance from the representative point of the cluster. Therefore, in this invention, it is proposed that the evaluation level is represented by a plurality of clusters divided in a vector space containing the plan vector data and the reference vector data, and that the evaluation level is calculated through calculation of the distribution state of the clusters. The evaluation level of the business plan is calculated from the distribution state of each of the clusters. For example, the K-means method can be used to classify vector data into a plurality of clusters (evaluation levels), and the vector data can be classified into each cluster (each evaluation level).

[0012] As described above, calculating the evaluation level for a business plan using plan vector data and reference vector data can be done relatively easily and with high accuracy by using various generative AIs, provided that appropriate prompts are created. For this reason, the present invention proposes that the evaluation level calculation unit is configured using a generative AI, and that the business plan and the evaluation criteria are provided to the generative AI as prompts.

[0013] Recently, one type of business plan that has been the subject of much debate due to the difficulty of evaluation is one that generates carbon credits. Carbon credits are a system that allows companies to buy and sell greenhouse gas emission reductions, primarily between companies. Companies quantify the amount of greenhouse gas reductions or absorptions generated by their environmental activities, and can trade emission rights certified as carbon credits with other companies. For this reason, carbon credit-related business plans require evaluation from multiple perspectives, and the number of evaluation criteria items is large, making the evaluation crucial. Furthermore, the evaluation criteria include those established in each country as well as the unique criteria and know-how of each company. Therefore, in this invention, the business plan is a business plan for acquiring carbon credits, and the evaluation criteria include general evaluation criteria created in accordance with carbon credit certification standards and unique evaluation criteria created independently. This allows business plans that generate carbon credits to be accurately evaluated with minimal burden.

[0014] Since carbon credits are traded between companies or brokers, the trading value of the carbon credits produced is important in business plans for generating carbon credits. For this reason, a high trading value of the carbon credits produced may also be included as an evaluation criterion. Accordingly, this invention proposes that the final evaluation of a business plan for acquiring carbon credits may be influenced by the trading value of the carbon credits.

[0015] The present invention also includes the scope of rights for an evaluation program that causes a computer to perform a process of evaluating a business plan using multiple evaluation criteria. Such an evaluation program causes a computer to perform a plan input process in which the business plan is input, a criteria input process in which the evaluation criteria are input, an evaluation level calculation process that calculates an evaluation level for each evaluation criterion for the business plan using the input business plan and evaluation criteria, and an evaluation process that performs a final evaluation of the business plan based on the evaluation levels. Through each process performed during its execution, the various effects and advantages described above for the evaluation system can be obtained.

[0016] Other features, operations, and effects of the present invention will be clarified by the description of the present invention using the following drawings.

Brief Description of the Drawings

[0017] [Figure 1] It is a schematic diagram showing the schematic configuration of the evaluation system. [Figure 2] It is a flowchart showing the flow of processing when a business plan is quantitatively evaluated. [Figure 3] It is a document diagram showing an example of a prompt. [Figure 4] It is an explanatory diagram explaining the classification of evaluation by the K-means method. [Figure 5] It is a list of evaluation levels for each evaluation criterion for a plurality of business plans. [Figure 6] It is a list of evaluation levels for a plurality of business plans.

Modes for Carrying Out the Invention

[0018] First, using FIG. 1, the schematic configuration of the evaluation system according to the present invention will be described. This evaluation system evaluates a business plan using a plurality of evaluation criteria. This system includes a plan input unit 11, a criterion input unit 12, an evaluation level calculation unit 20, and an evaluation unit 30.

[0019] The plan input unit 11 converts the input business plan into data that can be processed by the evaluation level calculation unit 20 and provides it to the evaluation level calculation unit 20. The criterion input unit 12 converts the evaluation criteria used when evaluating a business plan into data that can be processed by the evaluation level calculation unit 20 and provides it to the evaluation level calculation unit 20. The evaluation criteria have several to over a hundred items, and these items may be summarized in one or more documents.

[0020] The evaluation level calculation unit 20 is a core component of the evaluation system, which is substantially constructed using a computer unit. Using the given business plan document and evaluation criteria, it calculates the evaluation level for each evaluation criterion for the business plan document. For this reason, the evaluation level calculation unit 20 includes, as functional units, an instruction generation unit 21, a vector representation module 22, an arithmetic unit 23, and the like. Note that at least some of the processes in each component constituting the evaluation system are realized by a computer substantially by executing a computer program.

[0021] The instruction generation unit 21 (prompt generation unit 21) functions as an interface that receives an instruction from the user and generates an instruction script for the arithmetic unit 23 of this evaluation system. The instruction script for the arithmetic unit 23 generated by the instruction generation unit 21 includes the content of the user's instruction, the content of the business plan document, and the content of the evaluation criteria. Examples of the content of the user's instruction are "The input document is a report on a carbon credit project.", "For each criterion, find a description that serves as evidence for whether this project meets the criterion.", "Output the obtained evidence while strictly adhering to the output format.", and the like.

[0022] The vector representation module 22 generates plan document vector data by vector-representing the sentences (including words and phrases) constituting the business plan document included in the instruction script, and similarly generates criterion vector data by vector-representing the sentences constituting the evaluation criteria included in the instruction document. For the vector representation of sentences, for example, the Word·Embedding technique used in BERT (Bidirectional Encoder Representations from Transformers) and the like is used. The generated plan document vector data and criterion vector data are provided to the arithmetic unit 23 together with the content of the user's instruction.

[0023] The calculation unit 23 uses the plan vector data and the reference vector data, and, referring to user instructions, calculates the similarity score indicating the degree to which each evaluation criterion in the business plan is met, thereby calculating the evaluation level of the input business plan. The similarity score can be calculated using techniques such as vector operations (e.g., dot product operations) between each sentence in the plan vector data and each sentence in the reference vector data to determine the similarity between them. An evaluation level is calculated for each of the multiple criterion items included in the evaluation criteria. Here, the evaluation level can be considered as a collection of evaluation levels for each criterion item.

[0024] The evaluation unit 30 performs a final evaluation of the business plan being evaluated based on the evaluation level, which is the calculation result of the calculation unit 23, i.e., the output of the evaluation level calculation unit 20, and outputs the final evaluation of the business plan. The evaluation unit 30 may be incorporated into the evaluation level calculation unit 20 as one of its functional units.

[0025] The vector representation module 22 and the evaluation level calculation unit 20 may be integrated as a generation AI 25. In that case, the instruction generation unit 21 functions as a prompt generation unit 21. Furthermore, the functions of the evaluation unit 30 can also be incorporated into this generation AI 25. Alternatively, the evaluation unit 30 may be composed of an independent machine learning model, independent of the generation AI 25.

[0026] Next, one embodiment of the evaluation system of the present invention will be described. In this embodiment, the evaluation system evaluates a business plan for acquiring carbon credits based on carbon credit evaluation criteria. The evaluation level calculation unit 20 of this evaluation system includes a generating AI 25 that outputs an evaluation level when given a prompt generated by a prompt generation unit 21. The carbon credit evaluation criteria include general evaluation criteria created in accordance with carbon credit certification criteria and proprietary evaluation criteria created independently.

[0027] Carbon credits are emission allowances that companies and individuals can acquire by funding projects that reduce greenhouse gas emissions, and the value of these allowances is equivalent to the amount of greenhouse gas emissions reduced. Such greenhouse gas emission projects (hereinafter simply referred to as "projects") require significant investment, so it is important to carefully examine the project proposal in advance to ensure that the carbon credits generated by the project have appropriate value. Examples of project proposals include projects related to reducing greenhouse gas emissions from rice paddies, projects related to afforestation and deforestation control, and projects related to biochar, and such project proposals constitute the business plan in this invention. In the examples shown in Figures 3 and 6, a project proposal related to biochar is used as the business plan, but of course, the business plan in this invention is not limited to projects related to biochar.

[0028] Figure 2 shows the process flow when a business plan and a forestry business plan, which are project proposals related to biochar, are quantitatively evaluated by the evaluation system. This evaluation system can also sequentially evaluate various business plans using the same evaluation criteria. The following explanation will focus on the business plan, which is a project proposal related to biochar.

[0029] A business plan for the biochar project to be evaluated is created by the user. Simultaneously, the user creates carbon credit evaluation criteria (hereinafter simply referred to as "evaluation criteria") for evaluating this business plan. The created business plan and evaluation criteria are converted into text data and provided to the evaluation system.

[0030] The evaluation criteria consist of more than 100 items, some of which are as follows: (1) The raw materials must be pure biological waste biomass and must not be cultivated for any particular purpose. (2) The raw materials are left to rot or burn for purposes other than energy production. (3) Biochar is expressed in dry tonnage. Care must be taken to ensure that it does not contain moisture, as this will lead to an overestimation of the carbon sequestration capacity. (4) The biomass used shall not contain paint residue, solvents, or other potentially toxic impurities. (5) Raw materials must not be imported from other countries.

[0031] The evaluation system can generate prompts using a given business plan and evaluation criteria, but prompts may also be created by the user using the user interface of the prompt generation unit 21 and provided to the evaluation system. In this embodiment, the generation AI 25 is constructed using an LLM (Large-Scale Language Model), so prompts suitable for the specifications of this LLM are generated. An example of such a prompt is shown in Figure 3. In the example in Figure 3, the prompt generation unit 21 generates prompts based on instructions (user instructions), a business plan, and evaluation criteria provided by the user via the network or the like. Here, the prompts are written in English to improve the output accuracy of the generation AI 25. Of course, the prompts may also be written in Japanese.

[0032] If the amount of data in the input business plan is large, the document file name specified for loading from an appropriate memory location will be entered in the prompt. Also, if there are many items in the input evaluation criteria, prompts containing a predetermined number of evaluation criterion items will be sequentially given to the generation AI25, and the multiple output results output by the generation AI25 will be integrated.

[0033] The extent to which a business plan meets the evaluation criteria is determined based on the degree of similarity between the business plan and each evaluation criterion item (the degree to which the business plan satisfies the evaluation criteria). The evaluation level, which represents the degree to which the business plan satisfies each evaluation criterion, is set to the following six levels: "Yes", "Likely Yes", "Likely No", "No", "Irrelevant", and "Not Mentioned".

[0034] This evaluation system uses the similarity (distance) of each data point in the vector space of the vector data of the text constituting the business plan (plan vector data) and the vector data of the text constituting the evaluation criteria (evaluation criteria vector data) to make judgments based on similarity. In other words, each data point is classified into clusters based on the evaluation levels of "Yes," "Likely Yes," "Likely No," "No," "Irrelevant," and "Not Mentioned." That is, the classification of each vector data point to a specific cluster (specific evaluation level) is calculated from the distribution state of the vector data set for each cluster in the vector space. The K-means method is used for this classification.

[0035] In this embodiment, the K-means classification method, as shown in Figure 4, involves setting up six classification regions (corresponding to evaluation levels) corresponding to "Yes," "Likely Yes," "Likely No," "No," "Irrelevant," and "Not Mentioned" in a vector space where data sets consisting of plan vector data and reference vector data are scattered. The data sets are then classified into each of these classification regions, and the distance (Euclidean distance: L2distance) between the centroid of the data sets contained in each classification region and the centroid of each classification region is calculated. This distance provides quantitative support for the calculation of the evaluation level. To explain in more detail, for example, the classification is performed by the following procedure. (1) The description of each piece of evidence is converted into a vector representation to generate a set of vector data, (2) Cluster the vector data into each classification region, (3) Calculate the centroid vector of the cluster in each classification area, (4) For each piece of evidence, the classification region of the cluster with the lowest Euclidean distance among the centroid vectors calculated in (3) is used as the classification using the K-means method.

[0036] Therefore, the prompt will include output instructions such as six evaluation levels and a description of the evidence that underlies those evaluations.

[0037] The evaluation unit 30 assesses the value (quality) of the business plan being evaluated based on evaluation levels calculated for each of the more than 100 evaluation criteria, and generates evaluation data (evaluation report) that includes the carbon credit value of the business plan.

[0038] Figure 5 shows a reference example of a table output illustrating six evaluation levels across numerous evaluation criteria for multiple business plans. In this example, the data set targeted by the K-means method is vector data obtained by vectorizing the text (evidence) extracted by the generating AI as the basis for the evaluation calculation. Examples of text extracted as the basis for the evaluation calculation are as follows: (1) A certain company's charcoal production is done in batches. Firewood is packed into the kilns, the combustion chambers are sealed, and thermal decomposition is carried out in each kiln. During the cooling stage, heat is dissipated from the kiln walls and top, and the material cools down in that manner. After cooling, the combustion chambers are opened, and the carbonized material is processed for transport. (2) The moisture content of the biochar to prevent dust generation and explosions during transport is not mentioned. (3) This document does not contain details regarding the moisture content of the biochar to prevent dust generation and explosions during transport. (4) Moisture content of each large bag was measured to confirm that it was between 20-30%. (5) This document does not contain information regarding the transportation of biochar over distances exceeding 200 km and the associated greenhouse gas emissions.

[0039] In the table in Figure 5, each item in the vertical column indicates the document number that defines the business plan, and each item in the horizontal row indicates the evaluation criterion item number that defines the evaluation criterion item. For example, each cell in the second column displays the evaluation level for each evaluation criterion item for business plan document number 2, and the cell in row 2, column 2 (2,2) displays "Likely Yes" as the evaluation level for evaluation criterion item 2 for business plan document number 2.

[0040] Based on the evaluation level data shown in Figure 5, the evaluation unit 30 can also estimate the trading value of carbon credits for each business plan and generate a comprehensive evaluation report showing their value ranking.

[0041] In the table output shown as an example in Figure 6, the vertical columns are the same as in Figure 5. The first cell in each horizontal row (Document iD) shows the document number defining the business plan, the second cell in each horizontal row (labeled class) shows the evaluation level of the business plan, the third cell in each horizontal row (evidence) shows the text (evidence) extracted as the basis for calculating the evaluation, the fourth cell in each horizontal row (matched) shows whether the evaluation by the generating AI and the evaluation by the K-means method match (TRUE) or not match (FALSE), the fifth cell in each horizontal row (K-means classification) shows the evaluation level by the K-means method, and the sixth cell in each horizontal row (L2 distance) shows the distance value calculated by the K-means method. The fifth cell in each horizontal row shows the evaluation level that ranked first in the K-means method, and the sixth cell in each horizontal row shows the distance value that was the basis for ranking first. The cells from the seventh cell onward in each horizontal row show the evaluation levels and distance values ​​for second place and below in the K-means method, in order. The final evaluation of the business plan is the one that ranked first, and in Figure 6, the final evaluation of the business plan shown in document number 1 is "Yes". At that time, the distance values ​​to the evaluation levels from second place onwards are also shown, making it possible to quantitatively analyze the final evaluation.

[0042] [Another embodiment] (1) In the embodiments described above, the evaluation system was shown to be configured as an integrated system, but these components may be distributed across a network, and the evaluation system may be configured as a distributed system. For example, the prompt generation unit 21 of the evaluation level calculation unit 20 may be built on the user terminal, the generation AI 25 of the evaluation level calculation unit 20 and the evaluation unit 30 may be built on a remote cloud computing service, and the final evaluation results may be sent to the user terminal. Alternatively, the generation AI 25 may be built on a separate server or the user terminal.

[0043] (2) The evaluation unit 30 can be configured integrally with the generating AI 25, but it may also be constructed as a machine learning model that takes various data outputs from the generating AI 25 (for example, list data as shown in Figures 5 and 6) as input data and outputs a final result showing the value ranking of the business plan.

[0044] (3) In the embodiments described above, a business plan relating to the Biochar Project was mainly used as an example of a business plan, but the present invention is not limited thereto, and the evaluation system of the present invention can be applied to various business plans, such as energy conservation business plans and food self-sufficiency business plans, with similar advantages.

[0045] Furthermore, the configurations disclosed in the above embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with configurations disclosed in other embodiments, as long as no inconsistencies arise. Moreover, the embodiments disclosed herein are illustrative, and the embodiments of the present invention are not limited thereto, and can be modified as appropriate without departing from the object of the present invention. [Industrial applicability]

[0046] This invention can be applied to an evaluation system and evaluation program that evaluates a business plan using multiple evaluation criteria. [Explanation of Symbols]

[0047] 2: Evaluation Criteria Items 11: Plan Input Section 12: Reference Input Section 20: Evaluation level calculation unit 21: Instruction generation unit (prompt generation unit) 22: Vectorization Module 23: Arithmetic section 25: Generation AI 30: Evaluation Unit

Claims

1. An evaluation system that evaluates a business plan using multiple evaluation criteria, A plan input unit for entering the aforementioned business plan, A criteria input unit for inputting the aforementioned evaluation criteria, An evaluation level calculation unit that calculates the evaluation level for each evaluation criterion for the business plan using the input business plan and evaluation criteria, An evaluation unit that performs a final evaluation of the business plan based on the aforementioned evaluation level, An evaluation system equipped with the following features.

2. The evaluation system according to claim 1, wherein the evaluation level calculation unit generates plan vector data by representing the text constituting the business plan as vectors, generates reference vector data by representing the text constituting the evaluation criteria as vectors, and calculates the evaluation level based on the degree of similarity between the business plan and each of the evaluation criteria calculated using the plan vector data and the reference vector data.

3. The evaluation system according to claim 2, wherein the evaluation level is represented by a plurality of clusters separated in a vector space containing the plan vector data and the reference vector data, and the evaluation level is calculated through calculation of the distribution state of the clusters.

4. The evaluation system according to any one of claims 1 to 3, wherein the evaluation level calculation unit is configured using a generating AI, and the business plan and the evaluation criteria are provided to the generating AI as prompts.

5. The evaluation system according to claim 4, wherein the business plan is a business plan for acquiring carbon credits, and the evaluation criteria include general evaluation criteria created in accordance with carbon credit certification standards and proprietary evaluation criteria created independently.

6. The evaluation system according to claim 5, wherein the final evaluation of the business plan for obtaining the carbon credits is influenced by the trading value of the carbon credits.

7. An evaluation program that causes a computer to perform a process of evaluating a business plan using multiple evaluation criteria, The aforementioned business plan input process, Criteria input process for inputting the aforementioned evaluation criteria, An evaluation level calculation process that uses the input business plan and evaluation criteria to calculate the evaluation level for each evaluation criterion for the business plan, An evaluation process that performs a final evaluation of the business plan based on the aforementioned evaluation level, An evaluation program that executes the program.

Images