Evaluation system, evaluation program, and recording medium

Abstract

To enable a unified and easy evaluation of how accurately the information generated by language models describes a situation. [Solution] The evaluation system (100) is a system for evaluating the performance of a language model (M) that outputs information describing a traffic situation when an image of the situation is input from a camera (4) that takes pictures of the traffic situation, and comprises an acquisition unit (1) that acquires detection results generated by a sensor (5) different from the camera detecting the situation, a verification unit (2) that verifies the information based on the detection results, and an output unit (3) that outputs the verification results of the information.

Description

Technical Field

[0001] The present invention relates to an evaluation system, an evaluation program, and a recording medium.

Background Art

[0002] Non-Patent Document 1 describes determining the situation around an autonomous vehicle using an image obtained by photographing the surroundings of the autonomous vehicle with a camera and a large language model (LLM).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Current language models have difficulty accurately verbalizing the situation depicted in an image based solely on the image. For this reason, it was necessary for a person to evaluate while referring to the image how accurately the text generated by the language model verbalizes the situation. Also, when a person evaluates, the evaluation results may vary.

Means for Solving the Problems

[0005] To solve the above problems, an evaluation system according to one aspect of the present invention is an evaluation system for evaluating the performance of a language model that outputs information describing a traffic situation when an image of the traffic situation is input from a camera that has captured the traffic situation, and comprises an acquisition unit that acquires a detection result generated by a sensor different from the camera detecting the situation, a verification unit that verifies the information based on the detection result, and an output unit that outputs the verification result of the information.

[0006] Furthermore, an evaluation program according to another aspect of the present invention is an evaluation program that causes a computer to evaluate the performance of a language model that outputs information describing a traffic situation when an image of the situation is input from a camera that has taken a photograph of the situation, and causes the computer to execute an acquisition process to acquire detection results generated by a sensor different from the camera detecting the situation, a verification process to verify the information based on the detection results, and an output process to output the verification results of the information. Note that a computer-readable recording medium on which the evaluation program is recorded also falls within the scope of the present invention.

[0007] Furthermore, in another aspect of the present invention, a recording medium records information output by a language model that outputs information describing a traffic situation when an image of the traffic situation is input from a camera that has taken a photograph of the traffic situation, and comparative information describing the situation, which is generated by a verification unit that verifies the information based on detection results generated by a sensor different from the camera detecting the situation. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the functional configuration of an evaluation system according to an embodiment of the present invention. [Figure 2] This diagram illustrates an example of a sensor that generates detection results acquired by the acquisition unit of the system. [Figure 3] This diagram illustrates an example of how the verification unit of the system generates comparison words. [Figure 4] This diagram illustrates an example of how the verification unit of the system generates comparison words. [Figure 5] This table shows an example of the actual traffic situation and the verification results from the verification department. [Figure 6] This flowchart shows an example of a flow of an evaluation method according to an embodiment of the present invention. [Modes for carrying out the invention]

[0009] <Evaluation System 100> Hereinafter, an evaluation system 100 according to one embodiment of the present invention will be described in detail.

[0010] [Evaluation targets of the evaluation system] The evaluation system 100 is a system for evaluating the performance of language model M1. The language model M1 to be evaluated is a trained model constructed by machine learning using, for example, a set of images showing traffic conditions and sentences describing the content of those images as training data. The images showing traffic conditions are images obtained by cameras mounted on a vehicle (for example, an autonomous vehicle). The images may be of the outside or the inside of the vehicle. The images may also be videos or still images.

[0011] The language model M1 constructed in this manner outputs information describing a traffic situation when it receives an image of the situation from a camera that has captured the situation. This information includes text and audio expressed in natural language, as well as information expressed in formal language. The following explanation will use as an example a case in which the language model M1 is configured to output text describing the situation when it receives an image of the situation. The language model M1 according to this embodiment outputs text in a defined format. "Text in a defined format" refers to text that contains words that are essential elements for the text to be evaluated by the evaluation system 100. "Essential words" include at least the vehicle's driving situation. The vehicle's driving situation includes at least one of the following: the vehicle's position (e.g., intersection), the vehicle's intention to drive (e.g., right turn, left turn), the relative position of an object (e.g., ahead), the absolute position of an object (e.g., crosswalk), a causal object outside the vehicle (e.g., pedestrian), the action of the causal object (e.g., walking), and the vehicle's current action (e.g., stopped).

[0012] [Configuration of Evaluation System 100] As shown in Figure 1, the evaluation system 100 comprises an acquisition unit 1, a verification unit 2, and an output unit 3. The evaluation system 100 according to this embodiment further comprises a camera 4, a sensor 5, and a second camera 6.

[0013] [Camera 4] Camera 4 captures traffic-related conditions. In this embodiment, camera 4 is mounted on a vehicle V1. However, camera 4 does not necessarily have to be mounted on vehicle V1.

[0014] [Sensor 5] Sensor 5 is a sensor different from camera 4 (image sensor). Sensor 5 detects traffic-related situations and generates detection results. Sensor 5 includes, for example, a receiver of LiDAR, a receiver of radio detection and ranging (radar), a receiver of a satellite positioning system (GNSS), an inertial measurement unit (IMU), etc. Sensor 5 according to this embodiment is mounted on a single vehicle V1 together with camera 4. When sensor 5 is mounted on vehicle V1, sensor 5 includes at least any one of, for example, a position (distance / angle) sensor 51, a speed sensor 52, an acceleration sensor 53, a pressure sensor 54, a temperature sensor 55, a force (torque) sensor 56, a flow meter 57, and a gas sensor 58 as shown in FIG. 2.

[0015] The position (distance / angle) sensor 51 includes at least any one of a sensor 511 that detects the distance from an object existing in front, a sensor 512 that detects the distance from an object existing behind, a sensor 513 that detects the rotation angle of the steering wheel, a sensor 514 that detects the tilt angle of the throttle valve, a sensor 515 that detects the tilt angle of the accelerator pedal, and a sensor 516 that detects the tilt angle of the brake pedal.

[0016] The speed sensor 52 includes at least any one of a sensor 521 that detects the rotational speed of the wheel, a sensor 522 that detects the speed of the crankshaft, a sensor 523 that detects the speed of the camshaft, and a sensor 524 that detects the injection speed of the injection pump in a diesel engine.

[0017] The acceleration sensor 53 detects the acceleration (impact) acting on the vehicle body.

[0018] The pressure sensor 54 includes at least any one of a sensor 541 that detects the tire air pressure, a sensor 542 that detects the brake pressure, a sensor 543 that detects the hydraulic reservoir pressure in the power steering, a sensor 544 that detects the suction pressure, a sensor 545 that detects the filling pressure, a sensor 546 that detects the fuel pressure, a sensor 547 that detects the refrigerant pressure in the air conditioner, and a sensor 548 that detects the modulation pressure in the automatic transmission.

[0019] The temperature sensor 55 includes at least any one of a sensor 551 for detecting the tire temperature, a sensor 552 for detecting the intake air temperature, a sensor 553 for detecting the ambient temperature, a sensor 554 for detecting the internal temperature, a sensor 555 for detecting the evaporator temperature in the air conditioner, a sensor 556 for detecting the coolant temperature, and a sensor 557 for detecting the engine oil temperature.

[0020] The force (torque) sensor 56 includes at least any one of a sensor 561 for detecting the force of stepping on the pedal, a sensor 562 for detecting the weight of the occupant, a sensor 563 for detecting the torque acting on the drive shaft, and a sensor 564 for detecting the torque acting on the steering wheel.

[0021] The flow meter 57 includes at least one of a sensor 571 for detecting the fuel flow rate and the amount of fuel supplied to the engine, and a sensor 572 for detecting the amount of air sucked by the engine.

[0022] The gas sensor 58 includes at least one of a sensor 581 for detecting the composition of the exhaust gas and a sensor 582 for detecting harmful substances contained in the supplied air.

[0023] Note that the above various sensors 5 are known, as disclosed, for example, on the following web pages. ·Vehicle sensors functions and types https: / / innovationdiscoveries.space / vehicle-sensors-functions-and-types / ·Automotive sensors: the design engineer’s guide https: / / www.avnet.com / wps / portal / abacus / solutions / markets / automotive-and-transportation / automotive / communications-and-connectivity / automotive-sensors /

[0024] [Second Camera 6] The second camera 6 is a different camera from camera 4. The second camera 6 may be mounted on a single vehicle V1 together with camera 4 and sensor 5, or it may be installed on the road, or it may be mounted on another vehicle V2 different from vehicle V1.

[0025] [Acquisition part 1] The acquisition unit 1 acquires the detection results generated by the sensor 5 when it detects a situation. In this embodiment, the acquisition unit 1 acquires detection results from at least one of the following: a LiDAR receiver, a radar receiver, a GPS positioning system (GNSS) receiver, and an inertial measurement unit (IMU). If the sensor 5 is mounted on a vehicle V1, the acquisition unit 1 may acquire detection results from at least one of the following: a position sensor 51, a speed sensor 52, an acceleration sensor 53, a pressure sensor 54, a temperature sensor 55, a force sensor 56, a flow meter 57, and a gas sensor 58, instead of, or in addition to, the detection results from the LiDAR receiver, the radar receiver, the GPS positioning system receiver, and the inertial measurement unit. In this embodiment, the acquisition unit 1 further acquires a second image generated when the second camera 6 photographs the situation. In addition, the acquisition unit 1 in this embodiment further acquires route information indicating the driving route of the autonomous vehicle.

[0026] [Verification Section 2] The verification unit 2 verifies the text based on the detection results acquired by the acquisition unit 1 (generated by the sensor 5). Based on the detection results, the verification unit 2 generates comparative information explaining the situation in the same format as the information generated by the language model M1. As described above, the language model M1 in this embodiment generates text. Therefore, the verification unit 2 in this embodiment generates comparative text explaining the situation in the same format as the text generated by the language model M1, based on the detection results. In addition, the verification unit 2 in this embodiment may verify the text and generate comparative text based on the detection results and the second image acquired by the acquisition unit 1 (generated by the second camera 6). In addition, the verification unit 2 in this embodiment may verify the text and generate comparative text based on the detection results and path information.

[0027] For example, in generating a contrasting word corresponding to "vehicle driving intention," which is one of the contrasting words in a comparative text, the verification unit 2 estimates the position of the autonomous vehicle based on the detection results, as shown in Figure 3, and measures the state of the autonomous vehicle to estimate its operation. In this embodiment, the verification unit 2 stores the estimated position and operation of the autonomous vehicle in the database D. Then, the verification unit 2 generates a contrasting word indicating the autonomous vehicle's driving intention based on the autonomous vehicle's position and operation. Specifically, the verification unit 2 inputs the position and operation of the autonomous vehicle into a Kalman filter, rule-based AI, etc., and also inputs the driving route as needed to output the contrasting word.

[0028] Furthermore, in generating "the current operation of the vehicle," which is one of the comparison words in the comparison text, the verification unit 2 derives the position of the object based on the detection results, as shown in Figure 4. The position of the object includes at least one of the position of the causative object, the absolute position of the object, and the relative position of the object. In this embodiment, the verification unit 2 stores the derived position of the object in the database D. Then, the verification unit 2 generates comparison words that indicate the operation of the object by performing a time-series analysis of the position. Specifically, the verification unit 2 generates comparison words by performing a time-series analysis of the obtained position of the object.

[0029] After generating a comparison sentence, the verification unit 2 compares the essential words contained in the sentence with the corresponding comparison words in the comparison sentence. As shown in Figure 5, for example, suppose that in a real-world situation where a bicycle is riding on a crosswalk, the language model M1 generates a sentence such as "A person is walking on the crosswalk." In response, the verification unit 2 generates a comparison sentence, "A bicycle is riding on the crosswalk." By comparing each comparison word in this comparison sentence with each essential word contained in the sentence generated by the language model M1, the verification unit 2 can easily verify that there was an error in the language model M1's recognition of the causal object. Incidentally, in conventional technology, the language model M1 only generates the sentence "A person is walking on the crosswalk," so verification had to be done by a human.

[0030] [Output section 3] Output unit 3 outputs the results of the text verification. In this embodiment, output unit 3 outputs information about required words whose meaning differed from the comparison words as verification results. The information about required words includes the type of required word and whether the required word is correct or incorrect. Output includes input to a generative model that generates training data for fine-tuning the language model M1, transmission to other devices (control devices of autonomous vehicles, drives that write to recording media, etc.), and display on a display device. Output unit 3 may also be configured to output the evaluation score of the language model M and judgment results such as pass / fail.

[0031] The output unit 3 may be configured to write the text output by the language model M1 and the comparison information generated by the verification unit 2 to a recording medium. In other words, the output unit 3 may be a device that creates a recording medium on which the information output by the language model M1 and the comparison information generated by the verification unit 2 are recorded. In this case, the output unit 3 may be configured to further write to the recording medium information about essential words whose meaning differs from the comparison words, obtained as a result of comparing the essential words included in the information with the comparison words included in the comparison information. In other words, the output unit 3 may be a device that creates a recording medium on which information about essential words whose meaning differs from the comparison words is further recorded.

[0032] [Variations of evaluation system 100] Furthermore, when a single image is input, the language model M1 may output multiple types of sentences, each with different descriptions of the driving conditions. In this case, the acquisition unit 1 may be configured to acquire multiple types of sentences from the language model M1 and select the sentence from among the multiple types of explanatory sentences that has the greatest impact on the vehicle's future driving.

[0033] Furthermore, Figure 1 illustrates an evaluation system 100 in which the language model M1 is not included in the configuration. However, the language model M1 may be included in the evaluation system 100.

[0034] Furthermore, Figure 1 illustrates a case where text is directly supplied from the language model M1 to the acquisition unit 1. However, the evaluation system 100 may be configured to store the text output by the language model M1 in a memory unit not shown. The acquisition unit 1 may then be configured to indirectly acquire the text from the memory unit.

[0035] [Effects and benefits of evaluation system 100] The evaluation system 100 described above automatically compares the text (inference result) generated by the language model with the sensor detection result using the verification unit 2, thus eliminating the need for effort to find images (words) that the language model struggles with. Furthermore, because the verification unit 2 makes decisions mechanically according to the evaluation system 100, it can always determine correctness based on a consistent standard (the judgment is consistent). Therefore, the evaluation system 100 allows for a consistent evaluation of how accurately the text generated by the language model verbalizes the situation with minimal effort.

[0036] <Evaluation Method S100> Hereinafter, an evaluation method S100 according to another embodiment of the present invention will be described in detail.

[0037] [Evaluation Method S100 Flowchart] Evaluation method S100 is a method for evaluating the performance of a language model that outputs information describing a traffic situation when an image of the situation is input from a camera 4 that has captured the situation. As shown in Figure 6, evaluation method S100 includes an acquisition step S1, a verification step S2, and an output step S3.

[0038] [Acquisition Step S1] In acquisition step S1, the computer acquires the detection results generated by a sensor other than camera 4 detecting the situation. The acquisition of the detection results may be performed using the acquisition unit 1 of the evaluation system 100, or by other means.

[0039] [Verification Step S2] After obtaining the detection results, the process moves to verification step S2. In verification step S2, the computer verifies the information based on the detection results. The information may be verified using the verification unit 2 of the evaluation system 100 described above, or by other means.

[0040] [Output step S3] After verifying the information, the process moves to output step S3. In output step S3, the computer outputs the verification results of the information. The output of the verification results may be performed using the output unit 3 of the evaluation system 100 described above, or by other means.

[0041] [Effects of Evaluation Method S100] The evaluation method S100 described above allows the computer to automatically compare the information (inference results) generated by the language model with the sensor detection results in verification step S2, thus saving time and effort in finding images (words) that the language model struggles with. Furthermore, according to evaluation method S100, the computer makes a mechanical judgment in verification step S2, ensuring that the correctness of the judgment is always determined by a consistent standard (the judgment is consistent). Therefore, evaluation method S100 allows for a consistent evaluation of how accurately the information generated by the language model verbalizes the situation with minimal effort.

[0042] <Variation> The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0043] For example, each part of the evaluation system 100 is an evaluation program for causing a computer to function as each part, and each part can be realized by an evaluation program for causing a computer to function as each part. In this case, each part is equipped with a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the evaluation program. Each part (acquisition unit 1, verification unit 2, output unit 3) is realized by executing each process of the evaluation program (acquisition process, verification process, output process) using this control device and storage device. The evaluation system 100 may be configured so that one computer realizes one of the parts, or one computer may be configured to realize two or more of the parts.

[0044] Furthermore, the evaluation program may be recorded on one or more computer-readable recording media, rather than temporarily. These recording media may or may not be provided to each unit. In the latter case, the evaluation program may be supplied to each unit via any wired or wireless transmission medium.

[0045] Furthermore, some or all of the functions of each part can be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as each part are formed is also included in the scope of this invention. In addition, it is also possible to realize the functions of each part by, for example, a quantum computer.

[0046] 〔summary〕 An evaluation system according to aspect 1 of the present invention is an evaluation system for evaluating the performance of a language model that outputs information describing a traffic situation when an image of the traffic situation is input from a camera that has captured the traffic situation, and comprises an acquisition unit that acquires detection results generated by a sensor different from the camera detecting the situation, a verification unit that verifies the information based on the detection results, and an output unit that outputs the verification results of the information.

[0047] In the evaluation system according to aspect 2 of the present invention, the acquisition unit may be configured to acquire detection results from at least one of a LiDAR light receiver, a radio wave detection rangefinder receiver, a satellite positioning system receiver, and an inertial measuring device, as described in aspect 1 above.

[0048] An evaluation system according to embodiment 3 of the present invention may be configured such that, in embodiment 1 or 2 above, the verification unit generates comparative information explaining the situation based on the detection result in the same format as the information, compares the words included in the information with comparative words included in the comparative information that correspond to the words, and the output unit outputs information regarding the words whose meaning differs from the comparative words as a verification result.

[0049] An evaluation system according to aspect 4 of the present invention may be configured such that, in aspect 3 above, the information includes words indicating the driving intention of the autonomous vehicle, the verification unit estimates the position of the autonomous vehicle based on the detection result, measures the state of the autonomous vehicle to estimate the operation of the autonomous vehicle, and generates comparative words indicating the driving intention of the autonomous vehicle based on the position and operation of the autonomous vehicle.

[0050] The evaluation system according to aspect 5 of the present invention may be configured such that, in aspect 3 or 4 above, the information includes words indicating the movement of an object located outside the autonomous vehicle, and the verification unit derives the position of the object based on the detection result, and generates comparative words indicating the movement of the object by analyzing the position in a time series.

[0051] An evaluation system according to embodiment 6 of the present invention may be configured such that, in any of embodiments 1 to 5 above, the acquisition unit further acquires a second image generated by a second camera different from the camera capturing the situation, and the verification unit verifies the information based on the detection result and the second image.

[0052] The evaluation system according to aspect 7 of the present invention may further include the camera and the sensor in the above-described aspect 6, wherein both the camera and the sensor are mounted on a single vehicle.

[0053] The evaluation system according to embodiment 8 of the present invention may further include the second camera in embodiment 7 described above, wherein the second camera is mounted together with the camera and the sensor in a single vehicle.

[0054] The evaluation system according to aspect 9 of the present invention may further include the second camera in aspect 7 or 8 described above, wherein the second camera is installed on the road or mounted on another vehicle different from the vehicle.

[0055] The evaluation system according to aspect 10 of the present invention may be configured such that, in any of the above aspects 1 to 9, the acquisition unit further acquires route information indicating the driving route of the autonomous vehicle, and the verification unit verifies the information based on the detection result and the route information.

[0056] An evaluation program according to aspect 11 of the present invention is an evaluation program that causes a computer to evaluate the performance of a language model that outputs information describing a traffic situation when an image of the traffic situation is input from a camera that has taken a picture of the traffic situation, and the program is configured to cause the computer to execute an acquisition process that acquires detection results generated by a sensor different from the camera detecting the situation, a verification process that verifies the information based on the detection results, and an output process that outputs the verification results of the information.

[0057] The recording medium according to aspect 12 of the present invention has a configuration in which, when an image of a traffic situation is input from a camera that has taken a photograph of the situation, the recording medium stores information output by a language model that outputs information describing the situation, and comparative information describing the situation, which is generated by a verification unit that verifies the information based on detection results generated by a sensor different from the camera detecting the situation.

[0058] A recording medium according to aspect 13 of the present invention may be configured to further record information relating to a word that has a different meaning from the comparison word, obtained as a result of comparing the word included in the information with a comparison word included in the comparison information that corresponds to the word, in the above aspect 12. [Explanation of symbols]

[0059] 100 Evaluation System 1 Acquisition part 2. Verification Department 3. Output section 4 cameras 5 sensors 6. Second camera M1 Language Model S100 Evaluation Method S1 Acquisition Steps S2 Verification Step S3 Output Step

Claims

1. An evaluation system for evaluating the performance of a language model that outputs information describing a traffic situation when an image of that situation is input from a camera that has captured such a situation, An acquisition unit that acquires detection results generated by a sensor different from the aforementioned camera detecting the aforementioned situation, A verification unit that verifies the information based on the detection results, An output unit that outputs the verification results of the aforementioned information, Equipped with, Evaluation system.

2. The acquisition unit acquires detection results from at least one of a LiDAR light receiver, a radio wave detection rangefinder receiver, a satellite positioning system receiver, and an inertial measuring device. The evaluation system according to claim 1.

3. The verification unit, Based on the detection results, comparative information explaining the situation is generated in the same format as the aforementioned information. The words included in the aforementioned information are compared with the corresponding comparison words included in the aforementioned comparison information. The output unit outputs information regarding the word whose meaning differs from the comparison word as a result of the verification. The evaluation system according to claim 1.

4. The aforementioned information includes words indicating the driving intentions of the autonomous vehicle, The verification unit, Based on the detection results, the position of the autonomous vehicle is estimated, and the state of the autonomous vehicle is measured to estimate the operation of the autonomous vehicle. Based on the position and movement of the autonomous vehicle, a contrasting word indicating the driving intention of the autonomous vehicle is generated. The evaluation system according to claim 3.

5. The aforementioned information includes words that describe the actions of objects located outside the autonomous vehicle. The verification unit, Based on the detection results, the position of the object is derived. The aforementioned position is analyzed over time to generate comparative words that indicate the movement of the object. The evaluation system according to claim 3 or 4.

6. The acquisition unit further acquires a second image generated by a second camera, different from the first camera, capturing the situation. The verification unit verifies the information based on the detection result and the second image. The evaluation system according to claim 1.

7. The camera and the sensor further comprise The aforementioned camera and sensor are both mounted on a single vehicle. The evaluation system according to claim 6.

8. Further equipped with the aforementioned second camera, The second camera is mounted on a vehicle together with the camera and the sensor. The evaluation system according to claim 7.

9. Further equipped with the aforementioned second camera, The second camera is installed on the road or mounted on a vehicle other than the aforementioned vehicle. The evaluation system according to claim 7.

10. The acquisition unit further acquires route information indicating the driving route of the autonomous vehicle, The verification unit verifies the information based on the detection result and the route information. The evaluation system according to claim 1.

11. An evaluation program that allows a computer to evaluate the performance of a language model that outputs information describing a traffic situation when an image of that situation is input from a camera that has captured such a situation, To the aforementioned computer, An acquisition process that acquires a detection result generated by a sensor different from the aforementioned camera detecting the aforementioned situation, A verification process is performed to verify the information based on the detection results, Output processing to output the verification results of the aforementioned information, To execute Evaluation program.

12. When an image of a traffic situation is input from a camera that has captured such a situation, the language model outputs information that describes the situation, and Based on the detection results generated by a sensor different from the aforementioned camera detecting the situation, a verification unit that verifies the information generates comparative information explaining the situation, and It is recorded that Recording medium.

13. Further information is recorded regarding the word whose meaning differs from the comparison word, obtained as a result of comparing the word contained in the aforementioned information with the corresponding comparison word contained in the aforementioned comparison information. The recording medium according to claim 12.

Images