On-vehicle ai verification device, on-vehicle ai verification method, autonomous vehicle, and server

Abstract

Provided is an on-vehicle AI verification device capable of efficiently verifying performance quality of an on-vehicle object detection model. The on-vehicle AI verification device verifies performance of an on-vehicle AI on the basis of sensor data input from an on-vehicle sensor. The on-vehicle AI verification device comprises: a verification target AI execution unit that outputs first object information including an object class of an object in the sensor data by executing a verification target AI on the sensor data; a generative AI execution unit that outputs surrounding environment information including a caption of the object in the sensor data by executing a generative AI on the sensor data; a standardization processing unit that outputs second object information obtained by converting the caption in the surrounding environment information into an object class by using a dictionary; and an AI verification unit that verifies the verification target AI by comparing the object class of the object in the first object information with the object class of the object in the second object information.

Description

In-vehicle AI verification device, in-vehicle AI verification method, autonomous vehicle, and server

[0001] This invention relates to an in-vehicle AI verification device, an in-vehicle AI verification method, an autonomous vehicle, and a server for verifying the performance of in-vehicle AI installed in an autonomous vehicle.

[0002] In recent years, autonomous vehicles equipped with advanced driver-assistance systems (ADAS) that autonomously control the steering, drive, and braking systems of a vehicle based on the output of on-board sensors such as cameras, LiDAR, and radar, as well as autonomous driving systems, have been becoming increasingly common. ADAS is a general term for adaptive cruise control systems (ACC), advanced emergency braking systems (AEBS), lane keeping assist systems (LKAS), and others.

[0003] In some autonomous vehicles, image data captured by on-board cameras is analyzed using image analysis AI (Artificial Intelligence) to identify other vehicles, pedestrians, obstacles, traffic signs, etc., around the vehicle. Based on this analysis, the vehicle's steering system and other components are appropriately controlled to achieve desired advanced driver assistance and autonomous driving.

[0004] Patent Document 1 is an example of a document disclosing an image analysis AI for object detection used in such situations. For example, the abstract of this document states that, as a means of solving the problem of "training an object detection model using transfer learning," "a method 600 for training a machine learning model and updating a machine learning model trained to remove masked heads includes the steps of: generating training data for a machine learning model, which includes generating a training input including an image showing an object, and generating a target output for the training input, which includes bounding boxes, mask data, and class indications related to the indicated object; providing training data for training a machine learning model with respect to a set of training inputs including the generated training input and a set of target outputs including the generated target output; identifying a head of the trained machine learning model corresponding to predicting mask data for a given input image; and updating the machine learning model trained to remove the identified head."

[0005] Japanese Patent Publication No. 2023-43825

[0006] However, when using the technology described in Patent Document 1, the AI ​​model designers and others had to manually prepare the correct values ​​(training data) for the object class names of each object in the image, which were necessary for verifying the performance of the object detection model being trained. Therefore, verifying the performance of the object detection model using the technology described in Patent Document 1 required a considerable amount of work.

[0007] Therefore, the present invention aims to provide an in-vehicle AI verification device, an in-vehicle AI verification method, an autonomous vehicle, and a server that can efficiently verify the performance of an in-vehicle object detection model by providing the correct values ​​(training data) of the object class names of each object in images captured under various environments, without requiring manual intervention.

[0008] To solve the above problems, the present invention provides an in-vehicle AI verification device that verifies the performance of an in-vehicle AI based on sensor data input from an in-vehicle sensor, comprising: a verification target AI execution unit that outputs first object information including the object class of an object in the sensor data by executing a verification target AI on the sensor data; a generation AI execution unit that outputs ambient environment information including the caption of an object in the sensor data by executing a generation AI on the sensor data; a standardization processing unit that outputs second object information obtained by converting the caption in the ambient environment information into an object class using a dictionary; and an AI verification unit that verifies the verification target AI by comparing the object class of an object in the first object information with the object class of an object in the second object information.

[0009] According to the in-vehicle AI verification device, in-vehicle AI verification method, autonomous vehicle, and server of the present invention, the performance of an in-vehicle object detection model can be efficiently verified by providing the correct values ​​(training data) of the object class names of each object in images captured under various environments without manual intervention.

[0010] Hardware configuration diagram of the in-vehicle AI verification device of the present invention. Functional block diagram of the in-vehicle AI verification device of Example 1. Processing flowchart of the verification target AI execution unit of Example 1. Processing flowchart of the generation target AI execution unit of Example 1. Processing flowchart of the generation AI result processing unit of Example 1. Processing flowchart of the AI ​​verification unit of Example 1. Functional block diagram of the in-vehicle AI verification device of Example 2. Processing flowchart of the generation AI execution instruction unit of Example 2. Functional block diagram of the in-vehicle AI verification device of Example 3. Processing flowchart of the cloud transmission unit of Example 3. Functional block diagram of the in-vehicle AI verification device of Example 4. Processing flowchart of the class-based filtering unit of Example 4. Functional block diagram of the in-vehicle AI verification device of Example 5. Processing flowchart of the importance determination unit of Example 5. Processing flowchart of the importance-based filtering unit of Example 5.

[0011] Hereinafter, embodiments of the in-vehicle AI verification device of the present invention will be described with reference to the drawings.

[0012] <In-vehicle system 2> First, Figure 1 will be used to explain the hardware configuration for realizing the in-vehicle AI verification device 1 of the present invention. The illustrated in-vehicle system 2 is a system that is mainly installed in an autonomous vehicle to implement ADAS and an autonomous driving system, but by coordinating the various sensors 21, main ECU 22, sub-ECU 23, and auxiliary ECU 24 shown in the figure, it also functions as the in-vehicle AI verification device 1 of the present invention. Below, the outlines of each part of the in-vehicle system 2 will be explained sequentially, focusing on its relationship with the in-vehicle AI verification device 1.

[0013] <<Various Sensors 21>> The various sensors 21 are sensors for observing the surrounding environment of the vehicle, and are, for example, onboard cameras installed to capture images of the front, sides, and rear of the vehicle. The following explanation will use the case where the various sensors 21 are onboard cameras and the output of the various sensors 21 is image data as an example, but the various sensors 21 may also be radar or LiDAR. It goes without saying that if the various sensors 21 are radar or LiDAR, the main ECU 22, etc., which will be described later, should be configured to process the output of the radar or LiDAR.

[0014] <<Main ECU 22>> The main ECU 22 is an ECU (Electronic Control Unit) that is responsible for foreground processing that is constantly running when ADAS or autonomous driving systems are in use, and is equipped with a CPU (Central Processing Unit) 22a, a GPU (Graphics Processing Unit) 22b, memory 22c, and storage 22d. The CPU 22a and GPU 22b work together to execute various programs loaded from storage 22d, etc. into memory 22c, thereby realizing the AI ​​execution unit to be verified, which is a component of the in-vehicle AI verification device 1. In the following, we will omit explanations of well-known technologies in this type of computer technology field.

[0015] <<Sub-ECU 23>> The sub-ECU 23 is an ECU that handles background processing that operates as needed, and like the main ECU 22, it is equipped with a CPU 23a, GPU 23b, memory 23c, and storage 23d. The CPU 23a and GPU 23b work together to execute various programs loaded from storage 23d, etc., into memory 23c, thereby realizing the generation AI execution unit and other components of the in-vehicle AI verification device 1.

[0016] <<Auxiliary ECU 24>> The auxiliary ECU 24 is an ECU that assists the main ECU 22 and sub-ECU 23, and operates by receiving commands from the main ECU 22 and sub-ECU 23. Like the main ECU 22 and sub-ECU 23, the auxiliary ECU 24 is equipped with a CPU, GPU, memory, and storage, and various programs are executed collaboratively by the CPU and GPU to realize the functional unit which is a component of the in-vehicle AI verification device 1.

[0017] In Figure 1, the functions of the in-vehicle AI verification device 1 are distributed across multiple ECUs. This is to distribute the large computational load required to realize the in-vehicle AI verification device 1 across multiple CPUs and GPUs. Therefore, if CPUs and GPUs with sufficient computing power are used, it is also possible to configure the system so that all the functions of the in-vehicle AI verification device 1 are implemented in a single ECU. Furthermore, if the goal is to reduce the cost of the in-vehicle system, the functions of the sub-ECU 23 and auxiliary ECU 24 can be implemented on a server in the cloud, thereby reducing the number of in-vehicle ECUs.

[0018] Next, an embodiment 1 of the in-vehicle AI verification device of the present invention will be described using Figures 2 to 6.

[0019] <In-vehicle AI Verification Device 1> First, using the functional block diagram in Figure 2, the in-vehicle AI verification device 1 of this embodiment, which is implemented by the in-vehicle system 2 in Figure 1, will be described. As shown here, the in-vehicle AI verification device 1 comprises a verification target AI execution unit 11, a generation AI execution unit 12, a standardization processing unit 13, and an AI verification unit 14. The details of each unit will be described in order below.

[0020] <<AI Execution Unit 11 for Verification Target>> The AI execution unit 11 for verification target executes processing using a relatively small-scale / low-load AI model on the sensor data D1 acquired from various sensors 21, and outputs first object information I １ It is a functional unit. Hereinafter, the operation of the AI execution unit 11 for verification target when the AI for verification target is an object detection AI model will be described using the processing flowchart of FIG. 3. Note that since the AI execution unit 11 for verification target is a functional unit that outputs data necessary for realizing advanced driver assistance and autonomous driving in real time, the processing in FIG. 3 is always executed at a high frequency of about 30 times per second.

[0021] In step S11a, the AI execution unit 11 for verification target acquires sensor data D1 (image data) from various sensors 21 (in-vehicle cameras).

[0022] In step S11b, the AI execution unit 11 for verification target executes object detection processing on the image data using the object detection AI model, and extracts first object information I １ from the image. This first object information I １ is information that is necessary at any time for realizing appropriate advanced driver assistance and autonomous driving. Specifically, it is the object class of each object in the image and the coordinates of each object in the image. Here, the object class is, for example, a name representing the type of an object such as car, bike, traffic light, etc., and is the name selected as the most probable from the default name list of the object detection AI model.

[0023] In step S11c, the AI execution unit 11 for verification target outputs the first object information I １ extracted in step S11b to the AI verification unit 14.

[0024] Note that the first object information I １ output by the AI execution unit for verification target has the characteristic that it always has the same content if the combination of the object detection AI model used and the sensor data D1 is the same.

[0025] <<AI Generation Execution Unit 12>>The AI Generation Execution Unit 12 is a functional unit that executes processing using a relatively large-scale / high-load AI generation model on sensor data D1 acquired from various sensors 21 as necessary, and outputs ambient environment information. Hereinafter, the function of the AI Generation Execution Unit 12 when the AI for generation is an environmental recognition AI generation model will be described using the processing flowchart of FIG. 4. Note that since the AI Generation Execution Unit 12 is a functional unit that executes advanced calculations that take several seconds for the processing of each sensor data D1, unlike the Verification Target AI Execution Unit 11, it does not plan real-time processing of the sensor data D1.

[0026] In step S12a, the AI Generation Execution Unit 12 acquires sensor data D1 (image data) from various sensors 21 (in-vehicle cameras).

[0027] In step S12b, the AI Generation Execution Unit 12 executes environmental recognition processing on the image data using the environmental recognition AI generation model, and generates ambient environment information from the image. This ambient environment information is the original data of the correct value (teacher data) when evaluating the appropriateness of the object class in the first object information I output by the Verification Target AI Execution Unit 11. Specifically, it is the caption of each object in the image and the coordinates of each object in the image. Here, the caption is a sentence generated by the object detection generation AI model based on the input image data, and is, for example, a sentence representing the state of an object in the image such as "motorcar on the road" or "parked vehicle". １ It is information that is the original data of the correct value (teacher data) when evaluating the appropriateness of the object class in the first object information I. Specifically, it is the caption of each object in the image and the coordinates of each object in the image. Here, the caption is a sentence generated by the object detection generation AI model based on the input image data, and is, for example, a sentence representing the state of an object in the image such as "motorcar on the road" or "parked vehicle".

[0028] In step S12c, the AI Generation Execution Unit 12 outputs the ambient environment information generated in step S12b to the normalization processing unit 13.

[0029] Note that the ambient environment information output by the AI Generation Execution Unit 12 has the characteristic that even if the combination of the AI generation model used and the sensor data D1 is the same, the same content is not always output. Also, compared to the first object information I in which an object class selected from a predetermined name list is registered, the ambient environment information has the characteristic that various captions can be registered for the same object in the same state. １ It has the characteristic that various captions can be registered for the same object in the same state.

[0030] <<Standardization Processing Unit 13>> The standardization processing unit 13 executes processing by a large language model on the surrounding environment information acquired from the generation AI execution unit 12, and outputs the second object information I ２ It is a functional unit that does so. Hereinafter, the function of the standardization processing unit 13 will be described using the processing flowchart of FIG. 5

[0031] In step S13a, the standardization processing unit 13 acquires surrounding environment information from the generation AI execution unit 12

[0032] In step S13b, the standardization processing unit 13 performs natural language processing (morphological analysis, syntactic analysis) on the captions in the surrounding environment information using a large language model, and extracts noun words. For example, if the caption is "motorcar on the road", "motorcar" and "road" are extracted as noun words, and if the caption is "parked vehicle", "vehicle" is extracted as a noun word

[0033] In step S13c, the standardization processing unit 13 refers to the dictionary D2 and converts the noun words extracted in step S13b into more general names. For example, if "car" is registered in the dictionary D2 as a more general name such as "automobile", "auto motorcar", "vehicle", "bus", "truck", etc., in this step, the noun words "motorcar" and "vehicle" extracted in the previous step are converted into the object class name "car". As a result, even if the caption expression generated by the generation AI fluctuates for a specific object in the image, the object class name of the object can be unified by referring to the dictionary D2, and comparison with the object class name output by the verification target AI execution unit 11 becomes possible

[0034] In step S13d, the standardization processing unit 13 outputs the second object information I ２ that has converted the caption of the surrounding environment information into an appropriate object class

[0035] Through the above processing, the standardization processing unit 13 takes the surrounding environment information (caption, coordinates), which is the output of the environment recognition generation AI model, and the first object information I, which is the output of the object detection AI model. １ (Object class, coordinates) and second object information I in the same format ２ It can be converted to (object class, coordinates).

[0036] <<AI Verification Unit 14>> The AI ​​Verification Unit 14 processes the first object information I １ and second object information I ２ This is a function that verifies the performance of the AI ​​(object detection AI model) used in the AI ​​execution unit 11 by comparing the following. The following uses the processing flowchart in Figure 6 to verify the performance of the second object information I ２ The function of the AI ​​verification unit 14 when using the provided data as training data will be explained.

[0037] In step S14a, the AI ​​verification unit 14 receives the first object information I from the AI ​​execution unit 11 to be verified. １ Obtain it.

[0038] In step S14b, the AI ​​verification unit 14 receives the second object information I from the standardization processing unit 13. ２ Obtain it.

[0039] In step S14c, the AI ​​verification unit 14 processes the first object information I １ and second object information I ２ By comparing these, the performance of the AI ​​(object detection AI model) used in the AI ​​execution unit 11 under verification is verified.

[0040] One example of a verification method for this step is to first examine the first object information I. １ and second object information I ２ Extract objects with the same coordinates. Then, the object class name related to that object is the first object information I. １ and second object information I ２ The system then determines if they match. If the object class names of both objects match, the object detection by the object detection AI model being verified is judged as "correct." If they do not match, it is judged as "incorrect."

[0041] Another example of a verification method for this step is to first examine the first object information I. １ and second object information I ２ Extract objects with the same object class name. Then, the coordinates of those objects are obtained from the first object information I. １ and second object information I ２ The system then determines if the coordinates are approximately the same. If the coordinates of both objects are approximately the same, the object detection by the object detection AI model being verified is judged as "correct." If the coordinates of both objects are significantly different, it is judged as "incorrect."

[0042] Furthermore, if multiple objects are captured in the image, the first object information I described above will be provided for each object. １ and second object information I ２ It goes without saying that all that's needed is to compare and verify, in other words, to determine whether something is correct or incorrect.

[0043] In step S14d, the AI ​​verification unit 14 outputs the verification result from the previous step as AI verification result D3 to a desired storage (for example, storage 23d of the sub-ECU 23). The AI ​​verification result D3 includes sensor data D1 and first object information I １ , surrounding environment information, second object information I ２ The data may include any or all of the metadata of the sensor data D1 (imaging time, vehicle position at the time of imaging, and vehicle ID for identifying the individual autonomous vehicle).

[0044] Verification by the AI ​​verification unit 14 is performed each time surrounding environment information is generated (i.e., second object information I ２ This process is performed each time a result is generated, and the AI ​​verification result D3 is accumulated each time. Therefore, by repeatedly generating ambient environment information (original data for the correct answer) by the generation AI execution unit 12, it becomes possible to more accurately determine the performance of the AI ​​being verified under various environmental conditions.

[0045] <Effects of this embodiment> According to the in-vehicle AI verification device 1 of this embodiment described above, the correct values ​​(training data) of the object class names of each object in image data captured under various environments can be prepared without human intervention, so the performance of the in-vehicle object detection model can be efficiently verified.

[0046] Next, an embodiment 2 of the in-vehicle AI verification device of the present invention will be described using Figures 7 and 8. Note that repetitive explanations of points common to embodiment 1 will be omitted.

[0047] First, the in-vehicle AI verification device 1 of this embodiment, which is implemented by the in-vehicle system 2 of Figure 1, will be described using the functional block diagram of Figure 7. As shown here, the in-vehicle AI verification device 1 of this embodiment is a device that includes a generation AI execution instruction unit 15 in addition to the parts described in Embodiment 1 (the AI ​​execution unit 11 to be verified, the generation AI execution unit 12, the standardization processing unit 13, and the AI ​​verification unit 14), and is a device that receives system control information D4, driver control information D5, driving environment information D6, and generation AI execution conditions D7 in addition to the sensor data D1 described in Embodiment 1. Note that it is not necessary for all of the system control information D4, driver control information D5, and driving environment information D6 to be input to the in-vehicle AI verification device 1 of this embodiment; it is sufficient if at least one of them is input.

[0048] As described above, the AI ​​to be verified executed by the AI ​​execution unit 11 is a small-scale model, while the AI ​​to be generated executed by the AI ​​execution unit 12 is a large-scale model. Therefore, the computational processing power of the ECU of the in-vehicle system 2 is insufficient to process the first object information I of the small-scale model. １ One issue is that it is not possible to generate ambient environment information using a large-scale model at the same speed as generating the first object information I. On the other hand, there is also the issue that verification by the AI ​​verification unit 14 does not need to be processed in real time. Considering these circumstances, in this embodiment, the first object information I is generated using the time-series sensor data D1 stored in the ECU's storage. １ and second object information I ２ We enabled asynchronous generation, allowing for subsequent AI verification.

[0049] <<Generation AI Execution Instruction Unit 15>> The Generation AI Execution Instruction Unit 15 is a functional unit that determines the timing of AI verification execution by instructing the Generation AI Execution Unit 12 to start and stop the execution of the generation AI. The functions of the Generation AI Execution Instruction Unit 15 will be explained below using the processing flowchart in Figure 8.

[0050] In step S15a, the generated AI execution instruction unit 15 obtains system control information D4 from the ECU's storage. This system control information D4 is, for example, information relating to control commands to the steering system, drive system, braking system, etc., by ADAS or an autonomous driving system.

[0051] In step S15b, the generated AI execution instruction unit 15 obtains driver control information D5 from the ECU's storage. This driver control information D5 is, for example, information regarding manual operations by the driver, such as the steering wheel, accelerator pedal, and brake pedal.

[0052] In step S15c, the generation AI execution instruction unit 15 acquires driving environment information D6 from the ECU's storage. This driving environment information D6 is, for example, information about the driving environment such as the vehicle's current location, weather, and time of day.

[0053] In step S15d, the generation AI execution instruction unit 15 obtains the generation AI execution conditions D7 from the ECU's storage. These generation AI execution conditions D7 define the start and end conditions of the generation AI in relation to any of the system control information D4, driver control information D5, or driving environment information D6. If the vehicle is a so-called connected car, the generation AI execution conditions D7 may also be received via OTA (Over The Air).

[0054] In step S15e, the generation AI execution instruction unit 15 determines whether the generation AI execution unit 12 is currently executing the generation AI. If it is not currently executing, the process proceeds to step S15f; if it is currently executing, the process proceeds to step S15h.

[0055] In step S15f, the generated AI execution instruction unit 15 determines whether the conditions for starting the execution of the generated AI are met. If the conditions are met, the process proceeds to step S15g; otherwise, it returns to step S15a. Examples of execution start conditions include the following: (1) Execution start conditions related to system control information D4 include situations where a sudden steering control command to the steering system, a sudden acceleration control command to the drive system, or a sudden braking control command to the braking system is observed. (2) Execution start conditions related to driver control information D5 include situations where sudden operation of the steering wheel, accelerator pedal, brake pedal, etc. by the driver is observed. (3) Execution start conditions related to driving environment information D6 include situations where the vehicle's driving position changes from outside the tunnel to inside the tunnel, situations where the weather changes from sunny to cloudy or rainy, and situations where the time is sunset.

[0056] In step S15g, the generation AI execution instruction unit 15 outputs an instruction to start the generation AI execution to the generation AI execution unit 12. This starts the generation of ambient environment information by the generation AI execution unit 12.

[0057] On the other hand, in step S15h, the generation AI execution instruction unit 15 determines whether the execution termination conditions for the generation AI are met. If the conditions are met, the process proceeds to step S15i; otherwise, it returns to step S15a. Examples of execution termination conditions include the following: (1) Execution termination conditions related to system control information D4 include situations where sudden steering control commands to the steering system, sudden acceleration control commands to the drive system, or sudden braking control commands to the braking system, etc., are no longer observed. (2) Execution termination conditions related to driver control information D5 include situations where sudden operations by the driver, such as the steering wheel, accelerator pedal, or brake pedal, are no longer observed. (3) Execution termination conditions related to driving environment information D6 include situations where the vehicle's driving position changes from inside a tunnel to outside a tunnel, the weather changes from cloudy or rainy to sunny, or the time reaches sunrise.

[0058] In step S15i, the generation AI execution instruction unit 15 outputs a command to the generation AI execution unit 12 to terminate the generation AI execution. This completes the generation of ambient environment information by the generation AI execution unit 12. Therefore, in this embodiment, ambient environment information is generated and AI verification is performed based on only one of the following sensor data D1: (1) Sensor data D1 acquired during a period in which sudden steering, sudden acceleration, or sudden braking occurred by ADAS or an autonomous driving system. (2) Sensor data D1 acquired during a period in which sudden operation occurred by the driver. (3) Sensor data D1 acquired during any of the following periods: inside a tunnel, cloudy weather, rainy weather, or at night.

[0059] Through the above process, the AI ​​execution unit 12 generates ambient environment information for a limited time period, from when the predetermined execution start condition is met until when the execution end condition is met. Therefore, the AI ​​verification unit 14 generates first object information I for the same time period. １ and second object information I ２ By comparing these, the performance of the AI ​​(object detection AI model) used by the AI ​​execution unit 11 under verification can be verified retrospectively.

[0060] Next, an embodiment 3 of the in-vehicle AI verification device of the present invention will be described using Figures 9 and 10. Note that redundant explanations of points common to the above embodiment will be omitted.

[0061] First, using the functional block diagram in Figure 9, the in-vehicle AI verification device 1 of this embodiment, which is implemented by the in-vehicle system 2 in Figure 1, will be described. As shown here, the in-vehicle AI verification device 1 of this embodiment is a device that includes a verification result transmission unit 16 in addition to the parts described in Embodiment 1 (the AI ​​execution unit 11 to be verified, the AI ​​execution unit 12 to generate, the standardization processing unit 13, and the AI ​​verification unit 14), and is a device that can communicate with the cloud 3 via a wireless communication network.

[0062] The in-vehicle AI verification device 1 of the present invention determines the quality of the performance of the AI ​​under verification based on sensor data D1 actually acquired by various sensors 21 mounted on an autonomous vehicle. However, it was difficult to collect a wide variety of sensor data D1 without any omissions from a single autonomous vehicle. Therefore, in this embodiment, the AI ​​verification results D3 from multiple in-vehicle AI verification devices 1 are aggregated on a server in the cloud 3, enabling a more accurate and multifaceted evaluation of the performance of a specific AI under verification.

[0063] <<Verification Result Transmission Unit 16>> The verification result transmission unit 16 is a functional unit that transmits the AI ​​verification result D3, which is the output of the AI ​​verification unit 14, to a server on the cloud 3 or to in-vehicle storage. The functions of the verification result transmission unit 16 will be explained below using the processing flowchart in Figure 10.

[0064] In step S16a, the verification result transmission unit 16 obtains the AI ​​verification result D3 from the AI ​​verification unit 14.

[0065] In step S16b, the verification result transmission unit 16 determines whether it is connected to Cloud 3. If it is connected, it proceeds to step S16c; otherwise, it proceeds to step S16d.

[0066] In step S16c, the verification result transmission unit 16 transmits the AI ​​verification result D3 to the server on the cloud 3. As a result, the server on the cloud 3 can aggregate AI verification results D3 from a large number of autonomous vehicles, allowing the server on the cloud 3 to verify a specific AI target for verification with higher accuracy and from multiple perspectives.

[0067] On the other hand, in step S16d, the verification result transmission unit 16 temporarily stores the AI ​​verification result D3 in the vehicle's storage, such as the storage within the ECU. The AI ​​verification result D3 temporarily stored in the vehicle's storage can then be sent to the server on the cloud 3 once the connection to the cloud 3 is established.

[0068] Next, an embodiment 4 of the in-vehicle AI verification device of the present invention will be described using Figures 11 and 12. Note that repetitive explanations of points common to the above embodiment will be omitted.

[0069] First, using the functional block diagram in Figure 11, the in-vehicle AI verification device 1 of this embodiment, which is implemented by the in-vehicle system 2 in Figure 1, will be described. As shown here, the in-vehicle AI verification device 1 of this embodiment is a device that, in addition to the parts described in Embodiment 1 (the AI ​​execution unit 11 to be verified, the AI ​​execution unit 12 to generate, the standardization processing unit 13, and the AI ​​verification unit 14), is equipped with a class-based filtering unit 17 and filtering object class information D8.

[0070] In the in-vehicle AI verification device 1 of Example 1, the standardization processing unit 13 outputs the second object information I ２ The second object information I was input directly into the AI ​​verification unit 14, but ２ This could potentially include object class names that cannot be effectively utilized by the AI ​​verification unit 14. For example, the second object information I generated based on sensor data D1 (image data) of an airplane flying overhead. ２ This includes object class names corresponding to airplanes, such as "airplane," but since "airplane" is an object class name not registered in the name list of the AI ​​being verified (object detection AI model), it is information that cannot be effectively utilized by the AI ​​verification unit 14. Therefore, in the in-vehicle AI verification device 1 of this embodiment, filtered second object information I is created by filtering out object class names that cannot be used for AI verification in the AI ​​verification unit 14. ２ The value ' was to be input to the AI ​​verification unit 14. The function of the class-based filtering unit 17 will be explained below using the processing flowchart in Figure 12.

[0071] <<Class-based filtering unit 17>> In step S17a, the class-based filtering unit 17 receives the second object information I from the standardization processing unit 13. ２ Obtain it.

[0072] In step S17b, the class-based filtering unit 17 acquires filtering object class information D8 that has been prepared in advance by the AI ​​model designer or the like. The filtering object class information D8 may be a so-called whitelist for removing objects other than those with registered object class names, or a so-called blacklist for removing objects with registered object class names. Furthermore, if the vehicle is a so-called connected car, the filtering object class information D8 may be received via OTA (Over-the-Air).

[0073] In step S17c, the class-based filtering unit 17 filters the second object information I according to the filtering object class information D8. ２ Filter out information about objects with unnecessary object class names.

[0074] In step S17d, the class-based filtering unit 17 outputs the filtered second object information I2' to the AI ​​verification unit 14.

[0075] With the configuration of this embodiment, the AI ​​verification unit 14 only needs to perform the processing described in the flowchart of Figure 6 for objects with object class names necessary for AI verification. By eliminating unnecessary calculations in the AI ​​verification unit 14, the computational load on the ECU can be significantly reduced.

[0076] Next, an embodiment 5 of the in-vehicle AI verification device of the present invention will be described using Figures 13 to 15. Note that repetitive explanations of points common to the above embodiments will be omitted.

[0077] First, using the functional block diagram in Figure 13, the in-vehicle AI verification device 1 of this embodiment, which is implemented by the in-vehicle system 2 in Figure 1, will be described. As shown here, the in-vehicle AI verification device 1 of this embodiment is a device that, in addition to the parts described in Embodiment 1 (the AI ​​execution unit 11 to be verified, the AI ​​execution unit 12 to generate, the standardization processing unit 13, and the AI ​​verification unit 14), is equipped with an importance determination unit 18, an importance criterion filtering unit 19, and an importance threshold D9.

[0078] In the in-vehicle AI verification device 1 of Example 1, the ambient environment information output by the generation AI execution unit 12 was directly input to the standardization processing unit 13. However, this ambient environment information may contain captions that cannot be effectively utilized by the AI ​​verification unit 14. For example, a caption generated based on sensor data D1 (image data) of a display vehicle inside a car dealership building may contain sentences related to the display vehicle, such as "parked car in building." However, this caption is of low importance to ADAS and autonomous driving systems for autonomous control of the vehicle, and therefore cannot be effectively utilized by the AI ​​verification unit 14. Therefore, in the in-vehicle AI verification device 1 of this embodiment, filtered ambient environment information, which has had captions with low contribution to AI verification by the AI ​​verification unit 14 filtered in advance, is input to the standardization processing unit 13. The functions of the importance determination unit 18 and the importance criterion filtering unit 19 will be explained below using the processing flowcharts in Figures 14 and 15.

[0079] <<Importance Determination Unit 18>> In step S18a, the importance determination unit 18 acquires surrounding environment information from the generation AI execution unit 12.

[0080] In step S18b, the importance determination unit 18 grasps the state of each object in the sensor data D1 by processing the captions of the surrounding environment information in natural language, calculates the degree to which each object is involved in the control of the vehicle, and outputs the calculation result as importance. For example, another vehicle driving in front of the vehicle is assigned a high importance because it is highly involved in the control of the vehicle. On the other hand, display vehicles inside the car dealer building are assigned a low importance because they are less involved in the control of the vehicle. The importance assigned here is a score that is calculated by utilizing generative AI to determine the degree to which each object is involved in the control of the vehicle.

[0081] In step S18b, the importance determination unit 18 outputs the surrounding environment information and the importance determination result for each object within the surrounding environment information to the importance criterion filtering unit 19.

[0082] <<Importance Criteria Filtering Unit 19>> In step S19a, the importance criteria filtering unit 19 obtains surrounding environment information and the importance determination results for each object within that surrounding environment information from the importance determination unit 18.

[0083] In step S19b, the importance criterion filtering unit 19 acquires an importance threshold D9 that has been prepared in advance by the AI ​​model designer or the like.

[0084] In step S19c, the importance-based filtering unit 19 filters each object in the surrounding environment information based on the importance threshold D9. For example, if the importance threshold D9 is 50, filtered surrounding environment information is generated in which information about other vehicles driving in front of the vehicle, which is judged to have an importance of 90, is retained, and information about display vehicles inside the car dealer building, which is judged to have an importance of 10, is removed.

[0085] In step S19c, the importance criterion filtering unit 19 outputs the filtered ambient environment information generated in the previous step to the standardization processing unit 13.

[0086] With the configuration of this embodiment, the standardization processing unit 13 only needs to perform the processing described in the flowchart of Figure 5 for objects of importance necessary for AI verification. By eliminating unnecessary calculations in the standardization processing unit 13, the computational load on the ECU can be significantly reduced.

[0087] 1...In-vehicle AI verification device, 11...Verification target AI execution unit, 12...Generating AI execution unit, 13...Standardization processing unit, 14...AI verification unit, 15...Generating AI execution instruction unit, 16...Verification result transmission unit, 17...Class-based filtering unit, 18...Importance determination unit, 19...Importance-based filtering unit, 2...In-vehicle system, 21...Various sensors, 22...Main ECU, 23...Sub-ECU, 24...Auxiliary ECU, 3...Cloud

Claims

1. An in-vehicle AI verification device for verifying the performance of an in-vehicle AI based on sensor data input from an in-vehicle sensor, comprising: a verification target AI execution unit that outputs first object information including the object class of an object in the sensor data by executing a verification target AI on the sensor data; a generation AI execution unit that outputs ambient environment information including the caption of an object in the sensor data by executing a generation AI on the sensor data; a standardization processing unit that outputs second object information obtained by converting the caption in the ambient environment information into an object class using a dictionary; and an AI verification unit that verifies the verification target AI by comparing the object class of an object in the first object information with the object class of an object in the second object information.

2. An in-vehicle AI verification device according to claim 1, characterized in that the verification target AI execution unit, the generation AI execution unit, the standardization processing unit, and the AI ​​verification unit are mounted on an autonomous vehicle.

3. An in-vehicle AI verification device according to claim 1, characterized in that the AI ​​execution unit to be verified is mounted on an autonomous vehicle, and the generation AI execution unit, the standardization processing unit, and the AI ​​verification unit are located on a server in the cloud.

4. An in-vehicle AI verification device according to claim 1, further comprising a generation AI execution instruction unit that instructs the generation AI execution unit to start and end the execution of the generation AI, wherein the generation AI execution instruction unit instructs the generation AI execution unit to start the execution of the generation AI when at least one of the system control information, driver control information, or driving environment information satisfies a predetermined execution start condition, and instructs the generation AI execution unit to end the execution of the generation AI when at least one of the system control information, driver control information, or driving environment information satisfies a predetermined execution end condition.

5. An in-vehicle AI verification device according to claim 1, further comprising a verification result transmission unit for transmitting the verification results of the AI ​​verification unit to the cloud.

6. An in-vehicle AI verification device according to claim 1, further comprising a class-based filtering unit that filters objects in the second object information output by the standardization processing unit based on filtering object class information, wherein the second object information filtered by the class-based filtering unit is input to the AI ​​verification unit.

7. An in-vehicle AI verification device according to claim 6, characterized in that the filtering object class information is a whitelist for removing objects other than those with registered object class names, or a blacklist for removing objects with registered object class names.

8. An in-vehicle AI verification device according to claim 1, further comprising: an importance determination unit that determines the importance of objects in the surrounding environment information output by the generation AI execution unit; and an importance criterion filtering unit that filters the objects in the surrounding environment information based on a comparison result between their respective importance and an importance threshold.

9. An in-vehicle AI verification device according to any one of claims 1 to 8, characterized in that the in-vehicle sensor is an in-vehicle camera, the sensor data is image data, and the AI ​​to be verified is an object detection AI.

10. An in-vehicle AI verification device according to any one of claims 1 to 8, characterized in that the AI ​​to be verified is a relatively small-scale / low-load AI model, and the generated AI is a relatively large-scale / high-load generated AI model.

11. An in-vehicle AI verification method for verifying the performance of an in-vehicle AI based on sensor data input to an ECU from an in-vehicle sensor, comprising: a verification target AI execution step of executing a verification target AI on the sensor data to output first object information including the object class of an object in the sensor data; a generation AI execution step of executing a generation AI on the sensor data to output ambient environment information including the caption of an object in the sensor data; a standardization processing step of outputting second object information obtained by converting the caption in the ambient environment information into an object class using a dictionary; and an AI verification step of verifying the verification target AI by comparing the object class of an object in the first object information with the object class of an object in the second object information.

12. An autonomous vehicle characterized by comprising the standardization processing unit described in claim 1.

13. A server characterized by comprising the standardization processing unit described in claim 1.

Images