Evaluation system, evaluation program, and evaluation method

Abstract

To enable accurate and easy evaluation of a first language model that outputs a sentence describing the content of a traffic-related image when the image is input.SOLUTION: An evaluation system (100) evaluates the performance of a first language model and includes: a first evaluation unit configured to output a first evaluation value indicating the accuracy of a sentence based on a predetermined evaluation index; a second evaluation unit configured to use a second language model constructed through machine learning based on a traffic-related viewpoint and output a second evaluation value indicating the accuracy of a sentence; and a third evaluation unit configured to output an evaluation result of the first language model based on the first and second evaluation values. The second evaluation unit inputs, into the second language model, a correct answer sentence indicating the content that should be mainly interpreted from an image, together with a sentence and outputs, based on a first index derived from the content of the correct answer sentence and a second index derived from the content of the sentence, the second evaluation value indicating the degree of separation between the first and second indices, calculated by the second language model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , ,

[0005] , , ,

[0001] The present invention relates to an evaluation system, an evaluation program, and an evaluation method.

Background Art

[0002] In recent years, with the lifting of the ban on "Level 4" autonomous driving, demonstration experiments of autonomous vehicles have been carried out across the country. In autonomous driving, a huge amount of data is collected from the vehicle, and it is used for event recognition by the vehicle's AI, future prediction, driving plan, improvement (learning) of the AI, etc. For example, Non-Patent Document 1 below discloses the current situation of such demonstration experiments of autonomous vehicles.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

[0006] Furthermore, an evaluation program according to another aspect of the present invention is an evaluation program for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input, and causes a computer to perform a first evaluation process that outputs a first evaluation value indicating the accuracy of the text based on a predetermined evaluation index; a second evaluation process that outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, which is constructed by machine learning based on the perspective of traffic; and a third evaluation process that outputs the evaluation result of the first language model based on the first evaluation value and the second evaluation value, wherein in the second evaluation process, the computer is instructed to input to the second language model a correct answer text, which is text that shows the content that should be most read from the image, and the text itself, and the computer is instructed to output a numerical value as the second evaluation value that indicates the degree of deviation between the first index and the second index, based on a first index based on the content of the correct answer text and a second index based on the content of the text, calculated by the second language model. Note that a computer-readable recording medium that records a learning data generation program also falls within the scope of the present invention.

[0007] Furthermore, an evaluation method according to another aspect of the present invention is an evaluation method for evaluating the performance of a first language model that outputs a text describing the content of an image when an image related to traffic is input, and includes: a first evaluation step in which a computer outputs a first evaluation value indicating the accuracy of the text based on a predetermined evaluation index; a second evaluation step in which the computer outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, which is constructed by machine learning based on a traffic perspective; and a third evaluation step in which the computer outputs an evaluation result of the first language model based on the first evaluation value and the second evaluation value, wherein in the second evaluation step the computer inputs a correct answer text, which is a text that shows the content that should be most read from the image, and the text, and the computer outputs a numerical value indicating the degree of deviation between the first index and the second index as the second evaluation value, based on a first index based on the content of the correct answer text and a second index based on the content of the text calculated by the second language model. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the functional configuration of an evaluation system according to an embodiment of the present invention. [Figure 2] This diagram illustrates a modified example of the output method for the second evaluation value performed by the second evaluation unit of the system. [Figure 3] This diagram illustrates a modified example of the output method for the second evaluation value performed by the second evaluation unit of the system. [Figure 4] This diagram illustrates a modified example of the output method for the second evaluation value performed by the second evaluation unit of the system. [Figure 5] This flowchart shows an example of a flow of an evaluation method according to an embodiment of the present invention. [Modes for carrying out the invention]

[0009] <Evaluation System 100> Hereinafter, an evaluation system 100 according to one embodiment of the present invention will be described in detail.

[0010] [Evaluation targets of the evaluation system] The evaluation system 100 is a system for evaluating the performance of the first language model M1. The first language model M1 to be evaluated is a trained model constructed by machine learning using training data consisting of a set of traffic-related images and a sentence describing the content of the image. The traffic-related images are images obtained by cameras mounted on autonomous vehicles. The images may be of the outside or the inside of the vehicle. The images may also be videos or still images. When a new traffic-related image is input to the first language model M1, it outputs a sentence describing the content of the image.

[0011] [Configuration of the evaluation system] The evaluation system 100 evaluates the performance of the first language model M1 based on the text output by the first language model M1. As shown in Figure 1, the evaluation system 100 comprises a first evaluation unit 1, a second evaluation unit 2, and a third evaluation unit 3.

[0012] [First Evaluation Department 1] The first evaluation unit 1 outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators. The first evaluation unit 1 according to this embodiment includes a word evaluation unit 11, a context evaluation unit 12, and a calculation unit 13.

[0013] (Word evaluation section 11) The word evaluation unit 11 outputs a first numerical value based on the number of words in a given text that match the words in the corresponding correct answer text. The correct answer text is a sentence that describes the content of an image related to traffic input to the first language model M1, and is generated by means other than the first language model M1 (e.g., a person). In other words, the correct answer text is a sentence that shows the content that should be most easily read from the image. The correct answer text may be stored in a memory unit not shown, or it may be generated each time an evaluation is performed. In this embodiment, the word evaluation unit 11 outputs a first numerical value using an evaluation index such as "BLEU" or "BERTscore". The first numerical value output by the word evaluation unit 11 in this embodiment is a number greater than 0 and less than or equal to 1.

[0014] (Contextual evaluation unit 12) The context evaluation unit 12 outputs a second numerical value based on the similarity between the context of the sentence and the context of the correct answer sentence corresponding to that sentence. The word evaluation unit 11 according to this embodiment outputs a second numerical value using an evaluation index such as "sentenceBERT". The second numerical value output by the context evaluation unit 12 according to this embodiment is a number greater than 0 and less than or equal to 1.

[0015] (Calculation section 13) The calculation unit 13 calculates a first evaluation value based on the first and second numerical values. The first evaluation value is a statistical value of the first and second numerical values. The "statistical value" includes summation value, weighted summation value, mean, weighted mean, etc. The summation value is the sum of the first evaluation value and the second evaluation value. The mean is the average of the first evaluation value and the second evaluation value. The weighted mean is the average of the values ​​obtained by multiplying at least one of the first evaluation value and the second evaluation value by a weighting coefficient. The first evaluation value is a number greater than 0 and less than or equal to 1. In other words, the closer the first evaluation value is to 1, the more accurate the text is evaluated by the first evaluation unit 1.

[0016] [Second Evaluation Section 2] The second evaluation unit 2 uses the second language model M2 to output a second evaluation value indicating the accuracy of the text. The second language model M2 is a different language model from the first language model M1 and is a trained model constructed by machine learning based on the perspective of traffic. The second language model M2 used by the second evaluation unit 2 in this embodiment is a Large Language Model (LLM). Specifically, the second language model M2 consists of at least one of "BestScore," "Sentence Bert," and "GPT4." By using the second language model M2 configured in this way, the accuracy of the text (the performance of the first language model M1) can be evaluated from a human perspective. In other words, texts that could not be distinguished by word-based evaluation metrics because the words used are almost the same (for example, "Because the truck went straight, the car stopped suddenly" and "Because the car stopped suddenly, the truck went straight," which are texts with reversed causal relationships) can be distinguished as texts with different content. Furthermore, sentences that could not be distinguished by context-based evaluation metrics (for example, "The truck went straight, so the car came to a sudden stop" and "The truck went straight, so the car came to a slow stop," sentences with nearly identical contexts) can also be distinguished as sentences with different content. Note that the second language model M2 may be a trained model other than a large-scale language model. The second evaluation unit 2 comprises a human perspective evaluation unit 21 and a second calculation unit 22.

[0017] (Human Perspective Evaluation Department 21) The human perspective evaluation unit 21 inputs an article and a correct sentence corresponding to the article into the second language model M2. The human perspective evaluation unit 21 includes a first human perspective evaluation unit 211 and a second human perspective evaluation unit 212. The first human perspective evaluation unit 211 inputs the correct sentence into the second language model M2. The second human perspective evaluation unit 212 inputs the article into the second language model M2. The second language model M2 into which the article and the correct sentence are input calculates a first index and a second index. The first index is an index based on the content of the correct sentence. The first index calculated by the second language model M2 according to the present embodiment is a risk degree indicating the degree of risk of the event indicated by the correct sentence. The second index is an index based on the content of the article. The second index calculated by the second language model M2 according to the present embodiment is a risk degree indicating the degree of risk of the event indicated by the article. The first index and the second index output by the second language model M2 according to the present embodiment are numerical values greater than 0 and less than or equal to 1. Then, the first human perspective evaluation unit 211 acquires the first index calculated by the second language model M2. Also, the second human perspective evaluation unit 212 acquires the second index calculated by the second language model M2.

[0018] (Second calculation unit 22) Based on the first index and the second index calculated by the second language model M2, the second calculation unit 22 calculates a numerical value indicating the degree of divergence between the first index and the second index. Specifically, the second calculation unit 22 substitutes the first index and the second index into the following formula (1) to calculate a numerical value. Then, the second calculation unit 22 outputs the numerical value calculated by the second language model M2 as the second evaluation value. The second evaluation value output by the second calculation unit 22 is a numerical value greater than 0 and less than or equal to 1. That is, the closer the second evaluation value is to 1, the smaller the divergence between the content of the article and the correct sentence (the higher the accuracy of the article), which means that the second evaluation unit 2 evaluates it. Numerical value indicating the degree of divergence = 1.0 - |First index - Second index| ··· (1)

[0019] Furthermore, as shown in Figure 2, the second evaluation unit 2 may be configured to input one or more pairs of example sentences describing traffic-related events and risk levels indicating the degree of danger of the events described in the example sentences into the second language model M2 before inputting text into the second language model M2. In this way, by performing fusion learning, the second language model M2 will be able to calculate a first metric with higher accuracy. As a result, the second evaluation unit 2 will be able to output a second evaluation value with higher accuracy.

[0020] Furthermore, as shown in Figure 3, the second evaluation unit 2 may be configured to receive instructions (prompts) from the second language model M2 to evaluate images from a traffic safety perspective, in addition to the correct answer sentence and text. The instructions are to evaluate from the perspective of a remote monitor of the autonomous vehicle. In this case, the second evaluation unit 2 outputs a numerical value as the second evaluation value, indicating the degree of deviation between the first and second indicators, based on the instructions received by the second language model M2. This allows the second language model M2 to calculate the first indicator with greater accuracy. As a result, the second evaluation unit 2 can output a second evaluation value with greater accuracy.

[0021] Furthermore, the second evaluation unit 2 may be configured to output a second evaluation value using Retrieval Augmented Generation (RAG). In this case, the evaluation system 100 further includes a database 4, as shown in Figure 4. The database 4 is a collection of past texts describing the content of images related to past traffic, and stores multiple past texts with different content. The past texts include texts previously output by the first language model M1, texts describing the content of traffic accidents recorded by the police, and texts about traffic accidents reported in the past. When the second evaluation unit 2 inputs a text to the second language model M2, it extracts past texts similar in content to the text from among the multiple past texts stored in the database 4. The second evaluation unit 2 then inputs the extracted past texts along with the text to the second language model M2. In this way, the second language model M2 can calculate a first index with higher accuracy. As a result, the second evaluation unit 2 can output a second evaluation value with higher accuracy.

[0022] [Third Evaluation Section 3] The third evaluation unit 3 outputs an evaluation result of the first language model M1 based on the first evaluation value and the second evaluation value. In this embodiment, the third evaluation unit 3 outputs the product of the first evaluation value and the second evaluation value as the evaluation result. As described above, the first evaluation value and the second evaluation value are both numerical values ​​greater than 0 and less than or equal to 1. Therefore, the evaluation result, which is the product of the first evaluation value and the second evaluation value, is also a numerical value greater than 0 and less than or equal to 1, and the closer it is to 1, the higher the performance of the first language model M1.

[0023] [Variations of evaluation system 100] The first language model M1 may be constructed to output a formatted document. A "formatted document" refers to a document that contains essential elements. These elements include at least the vehicle's driving conditions (vehicle position, vehicle driving intention, objects, object positions, object movements, vehicle's current movements, etc.). Furthermore, the first language model M1 may output multiple types of sentences, each with different descriptions of the driving conditions, when a single image is input. In this case, the first evaluation unit 1 and the second evaluation unit 2 may be configured to obtain multiple types of sentences from the first language model M1 and select the sentence from among the multiple types of explanatory sentences that has the greatest impact on the vehicle's future driving.

[0024] Furthermore, Figure 1 illustrates an evaluation system 100 in which the first language model M1 and the second language model M2 are not included in the configuration. However, the evaluation system 100 may include at least one of the first language model M1 and the second language model M2.

[0025] Furthermore, Figure 1 illustrates a case where text is directly supplied from the first language model M1 to the first evaluation unit 1 and the second evaluation unit 2. However, the evaluation system 100 may be configured to store the text output by the first language model M1 in a memory unit not shown. The first evaluation unit 1 and the second evaluation unit 2 may then be configured to indirectly acquire the text from the memory unit.

[0026] [Effects and benefits of evaluation system 100] For example, the sentence "The truck went straight, so the car stopped abruptly" and the sentence "The car stopped abruptly, so the truck went straight" have reversed cause and effect relationships and are judged as completely different sentences from a human perspective. However, conventional word-based evaluation metrics could not distinguish between the two sentences because they use almost the same words. Similarly, the sentence "The truck went straight, so the car stopped abruptly" and the sentence "The truck went straight, so the car stopped slowly" describe different ways the car stopped and are judged as different sentences from a human perspective in terms of safety. However, conventional context-based evaluation metrics could not distinguish between the two sentences because their contexts are almost the same. In short, conventional methods had low accuracy in evaluating whether language models could correctly describe situations when high-risk events occurred, which is most important for the automation of remote monitoring. Therefore, it was still necessary for humans to verify whether the sentences generated by language models contained the information important for remote monitoring. However, human verification of sentences generated by language models and evaluation of language models are extremely time-consuming and costly. However, the evaluation system 100 described above has a third evaluation unit 3 that outputs an evaluation result for the first language model M1 by taking into account not only the first evaluation value, which indicates the structural accuracy of the text based on conventional evaluation indicators, but also the second evaluation value, which indicates the accuracy of the text as evaluated by the second language model M2 from the perspective of remote monitoring. For this reason, the evaluation system 100 makes it possible to accurately and easily evaluate the first language model M1 (confirming whether the text generated by the first language model M1 has extracted information that is important for remote monitoring).

[0027] <Evaluation Method S100> Hereinafter, an evaluation method S100 according to another embodiment of the present invention will be described in detail.

[0028] [Evaluation Method S100 Flowchart] Evaluation method S100 is a method for evaluating the performance of the first language model M1. As shown in Figure 5, evaluation method S100 includes a first evaluation step S1, a second evaluation step S2, and a third evaluation step S3.

[0029] [First Evaluation Step S1] In the first evaluation step S1, the computer outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators. The first evaluation value may be calculated using the first evaluation unit 1 of the evaluation system described above, or by other means.

[0030] [Second evaluation step S2] In the second evaluation step S2, the computer uses the second language model M2 to output a second evaluation value indicating the accuracy of the text. In the second evaluation step S2, the computer inputs the correct answer and the text into the second language model M2. Also in the second evaluation step S2, the computer outputs a numerical value as the second evaluation value indicating the degree of deviation between the first and second indicators calculated by the second language model M2. The calculation of the second evaluation value may be performed using the second evaluation unit 2 of the evaluation system described above, or by other means.

[0031] [Third evaluation step S3] In the third evaluation step S3, the computer outputs the evaluation result of the first language model M1 based on the first evaluation value and the second evaluation value. The output of the evaluation result may be performed using the third evaluation unit 3 of the evaluation system described above, or by other means.

[0032] [Effects of Evaluation Method S100] The evaluation method S100 described above outputs the evaluation result of the first language model M1 in the third evaluation step S3, taking into account not only the first evaluation value, which indicates the structural accuracy of the text based on conventional evaluation indicators, but also the second evaluation value, which indicates the accuracy of the text as evaluated by the second language model M2 from the perspective of remote monitoring. Therefore, according to evaluation method S100, similar to the evaluation system 100 described above, the evaluation of the first language model M1 (confirming whether the text generated by the first language model M1 has extracted information important for remote monitoring) can be performed accurately and easily.

[0033] <Variation> The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0034] For example, each part of the evaluation system 100 is a program that causes a computer to function as that part, and can be realized by a learning data generation program that causes a computer to function as that part. In this case, each part is equipped with a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the learning data generation program. Each part is realized by executing each process of the learning data generation program (first evaluation process, second evaluation process, third evaluation process) using this control device and storage device. The evaluation system 100 may be configured so that one computer realizes one of the parts, or one computer may be configured to realize two or more of the parts.

[0035] Furthermore, the learning data generation program described above may be recorded on one or more computer-readable recording media, rather than on a temporary basis. Each unit may or may not have such recording media. In the latter case, the program may be supplied to each unit via any wired or wireless transmission medium.

[0036] Furthermore, some or all of the functions of each part can be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as each part are formed is also included in the scope of this invention. In addition, it is also possible to realize the functions of each part by, for example, a quantum computer.

[0037] 〔summary〕 An evaluation system according to aspect 1 of the present invention is an evaluation system for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input, and comprises: a first evaluation unit that outputs a first evaluation value indicating the accuracy of the text based on a predetermined evaluation index; a second evaluation unit that outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, which is constructed by machine learning based on a traffic perspective; and a third evaluation unit that outputs the evaluation result of the first language model based on the first evaluation value and the second evaluation value, wherein the second evaluation unit is configured to input a correct answer sentence, which is text that shows the content that should be most read from the image, and the text, to the second language model, and outputs a numerical value as the second evaluation value indicating the degree of deviation between the first index and the second index based on a first index based on the content of the correct answer sentence and a second index based on the content of the text calculated by the second language model.

[0038] The evaluation system according to embodiment 2 of the present invention may be configured such that, in embodiment 1 above, the first evaluation unit includes a word evaluation unit that outputs a first numerical value based on the number of words in the text that match the number of words in the correct answer sentence; a context evaluation unit that outputs a second numerical value based on the similarity between the context of the text and the context of the correct answer sentence; and a calculation unit that calculates the first evaluation value based on the first numerical value and the second numerical value.

[0039] The evaluation system according to embodiment 3 of the present invention may be configured such that, in embodiment 2 described above, the first evaluation value and the second evaluation value are each numerical values ​​greater than 0 and less than or equal to 1, the first evaluation value is a statistical value of the first numerical value and the second numerical value, and the third evaluation unit outputs the product of the first evaluation value and the second evaluation value as the evaluation result.

[0040] The evaluation system according to aspect 4 of the present invention may be configured such that, in any of aspects 1 to 3 above, the first indicator is a risk level indicating the degree of danger of the event indicated by the correct answer statement, and the second indicator is a risk level indicating the degree of danger of the event indicated by the sentence.

[0041] The evaluation system according to aspect 5 of the present invention is configured such that, in aspect 4 above, the second language model is a large-scale language model, and before inputting the text into the second language model, the second evaluation unit inputs one or more sets of example sentences describing traffic-related events and a degree of risk indicating the degree of danger of the events shown in the example sentences into the second language model.

[0042] An evaluation system according to aspect 6 of the present invention may be configured such that, in any of the above aspects 1 to 5, it includes a database that stores multiple past texts with different contents that describe the contents of images related to past traffic, and when the second evaluation unit inputs the text into the second language model, it extracts past texts that are similar in content to the text from among the multiple past texts, and inputs the extracted past texts into the second language model together with the text.

[0043] The evaluation system according to embodiment 7 of the present invention may be configured such that, in any of embodiments 1 to 6 described above, the second evaluation unit inputs an instruction to the second language model to evaluate the image from the standpoint of traffic safety, in addition to the correct answer sentence and the text, and the second language model outputs a numerical value indicating the degree of discrepancy between the first index and the second index calculated based on the instruction as the second evaluation value.

[0044] The evaluation system according to aspect 8 of the present invention may be configured such that, in aspect 7 above, the traffic-related image is an image obtained by a camera mounted on an autonomous vehicle, and the instruction is an instruction to evaluate from the perspective of a remote monitor of the autonomous vehicle.

[0045] An evaluation program according to aspect 9 of the present invention is an evaluation program for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input, and is configured to cause a computer to perform a first evaluation process that outputs a first evaluation value indicating the accuracy of the text based on a predetermined evaluation index, a second evaluation process that uses a second language model different from the first language model and constructed by machine learning based on a traffic perspective to output a second evaluation value indicating the accuracy of the text, and a third evaluation process that outputs the evaluation result of the first language model based on the first evaluation value and the second evaluation value, wherein in the second evaluation process the computer is input to the second language model the correct answer text, which is text that shows the content that should be read most from the image, and the text, and the second language model outputs a numerical value indicating the degree of deviation between the first index and the second index as the second evaluation value, based on the first index based on the content of the correct answer text and the second index based on the content of the text calculated by the second language model.

[0046] An evaluation method according to aspect 10 of the present invention is an evaluation method for evaluating the performance of a first language model that outputs a text describing the content of an image when an image related to traffic is input, and includes: a first evaluation step in which a computer outputs a first evaluation value indicating the accuracy of the text based on a predetermined evaluation index; a second evaluation step in which the computer outputs a second evaluation value indicating the accuracy of the text using a second language model different from the first language model, which is constructed by machine learning based on a traffic perspective; and a third evaluation step in which the computer outputs an evaluation result of the first language model based on the first evaluation value and the second evaluation value, wherein in the second evaluation step the computer inputs a correct answer text, which is a text that shows the content that should be most read from the image, and the text, and the computer outputs a numerical value indicating the degree of deviation between the first index and the second index as the second evaluation value, based on a first index based on the content of the correct answer text and a second index based on the content of the text calculated by the second language model. [Explanation of Symbols]

[0047] 100 Evaluation System 1. First Evaluation Department 11 Word Evaluation Section 12 Contextual Evaluation Section 13 Calculation Section 2. Second Evaluation Department 21. Human Perspective Evaluation Department 22 Second calculation section 3. Third Evaluation Department 4 Databases M1 First Language Model M2 Second Language Model S100 Evaluation Method S11 First Evaluation Step S12 Second Evaluation Step S13 Third Evaluation Step

Claims

1. An evaluation system for evaluating the performance of a first language model that, when an image related to traffic is input, outputs text describing the content of the image, A first evaluation unit outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators, A second evaluation unit that uses a second language model different from the first language model, which is constructed using machine learning based on the perspective of traffic, and outputs a second evaluation value indicating the accuracy of the text, A third evaluation unit outputs an evaluation result of the first language model based on the first evaluation value and the second evaluation value, Equipped with, The second evaluation unit is, The second language model is given the correct answer sentence, which is the sentence that best represents the content to be read from the image, and the sentence itself. Based on the first indicator calculated by the second language model, which is based on the content of the correct answer sentence, and the second indicator calculated by the second language model, a numerical value indicating the degree of deviation between the first indicator and the second indicator is output as the second evaluation value. Evaluation system.

2. The aforementioned first evaluation unit is A word evaluation unit outputs a first numerical value based on the number of words in the aforementioned text that match the number of words in the aforementioned correct answer text, A context evaluation unit that outputs a second numerical value based on the similarity between the context of the aforementioned sentence and the context of the aforementioned correct answer sentence, A calculation unit that calculates the first evaluation value based on the first numerical value and the second numerical value, including, The evaluation system according to claim 1.

3. The first evaluation value and the second evaluation value are each a number greater than 0 and less than or equal to 1. The first evaluation value is a statistical value of the first numerical value and the second numerical value, The third evaluation unit outputs the product of the first evaluation value and the second evaluation value as the evaluation result. The evaluation system according to claim 2.

4. The first indicator is a risk level that indicates the degree of danger of the event described in the correct answer statement. The second indicator is a risk level that indicates the degree of danger of the event described in the preceding text. The evaluation system according to claim 1.

5. The aforementioned second language model is a large-scale language model, Before inputting the text into the second language model, the second evaluation unit inputs one or more sets into the second language model: an example sentence describing a traffic-related event, and a risk level indicating the degree of danger of the event described by the example sentence. The evaluation system according to claim 4.

6. This is a historical document describing the content of images related to past transportation, and it includes a database that stores multiple historical documents with different content. The second evaluation unit is, When inputting the aforementioned text into the second language model, past texts that are similar in content to the aforementioned text are extracted from among multiple past texts. The extracted past text is input into the second language model along with the text. The evaluation system according to claim 1.

7. The second evaluation unit is, The second language model is given instructions to evaluate the image from a traffic safety perspective, in addition to the correct answer sentence and the text. The second language model outputs a numerical value as the second evaluation value that indicates the degree of deviation between the first indicator and the second indicator calculated based on the instructions. The evaluation system according to claim 1.

8. The aforementioned traffic images are images obtained by cameras mounted on autonomous vehicles. The aforementioned instruction is an instruction to evaluate the autonomous vehicle from the perspective of a remote monitor. The evaluation system according to claim 7.

9. An evaluation program for evaluating the performance of a first language model that, when an image related to traffic is input, outputs a text describing the content of the said image, On the computer, A first evaluation process that outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators, A second evaluation process that uses a second language model different from the first language model, which is constructed using machine learning based on the perspective of traffic, and outputs a second evaluation value indicating the accuracy of the text, A third evaluation process outputs the evaluation result of the first language model based on the first evaluation value and the second evaluation value, Make it run, In the second evaluation process, the computer, The second language model is given the correct answer sentence, which is the sentence that best represents the content to be read from the image, and the sentence itself. Based on the first indicator calculated by the second language model, which is based on the content of the correct answer sentence, and the second indicator calculated by the second language model, a numerical value indicating the degree of deviation between the first indicator and the second indicator is output as the second evaluation value. Evaluation program.

10. An evaluation method for evaluating the performance of a first language model that outputs text describing the content of an image when an image related to traffic is input, A first evaluation step in which a computer outputs a first evaluation value indicating the accuracy of the text based on predetermined evaluation indicators, A second evaluation step in which a computer uses a second language model, which is different from the first language model and is constructed by machine learning based on the perspective of traffic, to output a second evaluation value indicating the accuracy of the text, A third evaluation step in which the computer outputs an evaluation result of the first language model based on the first evaluation value and the second evaluation value, Includes, In the aforementioned second evaluation step, The computer inputs the correct answer sentence, which is a sentence that best represents the content to be read from the image, and the sentence into the second language model. The computer outputs a numerical value as the second evaluation value, which indicates the degree of discrepancy between the first indicator and the second indicator, based on the first indicator calculated by the second language model, which is based on the content of the correct answer sentence, and the second indicator, which is based on the content of the sentence. Evaluation method.

Images