Update verification methods, update verification systems, update verification programs

Abstract

To provide an updating verification method which can specify that abnormal updating is performed at an early stage, when a machine learning model is upgraded as updating of a vehicle control device.SOLUTION: An updating verification method allows a computer to execute: acquisition of a first verification number indicating the number of objects around a vehicle recognized during a predetermined period or with a predetermined distance, or the number of driving determinations in a generated travel plan by an upgraded machine learning model; acquisition of a second verification number indicating the number of objects around a vehicle recognized during a predetermined period or with a predetermined distance, or the number of driving determinations in a generated travel plan by a machine learning model of a version prior to upgrading, by referring to log data; and verification whether or not the updating is normal by comparison of the first verification number with the second verification number.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a technique for verifying an update of a vehicle control device. In particular, it relates to a control device that performs automatic driving control of a vehicle using a machine learning model.

Background Art

[0002] In recent years, various techniques for utilizing a machine learning model as artificial intelligence (AI: Artificial Intelligence) in the field of vehicle control have been proposed. For example, as a document that discloses a technique for utilizing a machine learning model in the field of vehicle control, there is the following Patent Document 1. Also, as documents indicating the technical level of this technical field, there are the following Patent Documents 2 to 4.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] In vehicle control, particularly in autonomous driving control, improvements in control performance are expected through the use of machine learning models in each of the functions of perception, judgment, and operation. When machine learning models are used, updates to the control system that performs autonomous driving control may involve upgrading the machine learning model. It is envisioned that the new version of the machine learning model will be distributed to each vehicle subject to the update after its performance has been confirmed through offline verification and evaluation using experimental vehicles.

[0005] However, in machine learning models, inference results can change significantly with even small input changes. Furthermore, each vehicle is expected to have different sensor states and other characteristics. Therefore, in some vehicles, the updated machine learning model may not function correctly, potentially resulting in an unsuccessful update.

[0006] One of the purposes of this disclosure is to address the above-mentioned challenges and to provide a technology that enables early detection of abnormal updates when upgrading machine learning models. [Means for solving the problem]

[0007] The first aspect of this disclosure relates to an update verification method for verifying an update to a control device that performs automated driving control of a vehicle using a machine learning model, using a computer. The computer is configured to have access to one or more storage devices that store log data related to the inference results of the machine learning model. The update also includes upgrading the machine learning model. The machine learning model performs at least one of the following: recognition of the surrounding conditions of the vehicle or generation of a driving plan for automated driving control.

[0008] The update verification method for the first aspect involves a computer performing the following steps: obtaining a first verification number indicating the number of objects around the vehicle recognized by the upgraded machine learning model over a predetermined period or distance, or the number of driving decisions in the generated driving plan; obtaining a second verification number indicating the number of objects around the vehicle recognized by the pre-update version of the machine learning model over a predetermined period or distance, or the number of driving decisions in the generated driving plan, by referring to log data; and verifying whether the update is successful by comparing the first verification number and the second verification number.

[0009] A second aspect of this disclosure relates to an update verification system for verifying updates to a control device that performs autonomous driving control of a vehicle using a machine learning model. The update includes upgrading the machine learning model. The machine learning model performs at least one of the following: recognizing the surrounding environment of the vehicle or generating a driving plan for autonomous driving control.

[0010] The update verification system relating to the second aspect includes one or more processors and one or more storage devices for storing log data relating to the inference results of a machine learning model. The one or more processors are configured to perform the following processes: obtaining a first verification number indicating the number of objects around the vehicle recognized by the upgraded machine learning model over a predetermined period or distance, or the number of driving decisions in the generated driving plan; obtaining a second verification number indicating the number of objects around the vehicle recognized by the pre-update version of the machine learning model over a predetermined period or distance, or the number of driving decisions in the generated driving plan, by referring to the log data; and verifying whether the update is successful by comparing the first verification number and the second verification number.

[0011] The third aspect of this disclosure relates to an update verification program that causes a computer to perform an update to a control device that performs automated driving control of a vehicle using a machine learning model. The update verification program relating to the third aspect is configured to cause a computer to perform the update verification method relating to the first aspect. [Effects of the Invention]

[0012] According to this disclosure, the update of the control unit can be verified without requiring special or additional verification. This makes it possible to identify early on if an update has been performed incorrectly. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example configuration related to the automatic driving control of a vehicle. [Figure 2] This figure shows an example of the configuration of the control device according to this embodiment. [Figure 3] This figure shows the configuration of the update verification system according to this embodiment. [Figure 4] This is a diagram illustrating the update verification method according to this embodiment. [Figure 5] This flowchart shows the process executed by the processor according to this embodiment. [Figure 6] This diagram illustrates the comparison between the first and second verification numbers. [Modes for carrying out the invention]

[0014] 1. Control device The update verification method according to this embodiment verifies updates to a control device that performs automated driving control of a vehicle using a machine learning model. The control device targeted by the update verification method according to this embodiment will be described below.

[0015] FIG. 1 is a diagram showing a configuration example related to the automatic driving control of a vehicle 1 performed by a target control device. Automatic driving means automatically performing at least one of steering, acceleration, and deceleration of the vehicle 1 without depending on an operation by an operator. The automatic driving control is a concept that includes not only fully automatic driving control but also risk avoidance control, lane keep assist control, and the like. The operator may be a driver who rides in the vehicle 1 or a remote operator who remotely operates the vehicle 1.

[0016] The vehicle 1 includes a sensor group 10, a recognition unit 20, a planning unit 30, a control amount calculation unit 40, and a driving device 50.

[0017] The sensor group 10 includes a recognition sensor 11 used to recognize the situation around the vehicle 1. Examples of the recognition sensor 11 include a camera, LIDAR (Laser Imaging Detection and Ranging), a radar, and the like. The sensor group 10 may further include a state sensor 12 that detects the state of the vehicle 1, a position sensor 13 that detects the position of the vehicle 1, and the like. Examples of the state sensor 12 include an acceleration sensor, a yaw rate sensor, and the like. Examples of the position sensor 13 include a GNSS (Global Navigation Satellite System) sensor.

[0018] The sensor detection information SEN is information obtained by the sensor group 10. For example, the sensor detection information SEN includes an image captured by a camera. As another example, the sensor detection information SEN may include point cloud information obtained by LIDAR. The sensor detection information SEN may include vehicle state information indicating the state of the vehicle 1. The sensor detection information SEN may include position information indicating the position of the vehicle 1.

[0019] The recognition unit 20 receives sensor detection information SEN. Based on the information obtained by the recognition sensor 11, the recognition unit 20 recognizes the surrounding environment of the vehicle 1. For example, the recognition unit 20 recognizes objects around the vehicle 1. Examples of objects include pedestrians, other vehicles, white lines, road structures, fallen objects, traffic lights, intersections, signs, etc. The recognition result information RES indicates the recognition result by the recognition unit 20. For example, the recognition result information RES includes object information indicating the relative position and relative velocity of the object with respect to the vehicle 1.

[0020] The planning unit 30 receives recognition result information RES from the recognition unit 20. The planning unit 30 may also receive vehicle status information, location information, and map information. Based on the received information, the planning unit 30 generates a driving plan for vehicle 1. The driving plan may be for reaching a pre-set destination or for avoiding risks. The driving plan includes driving decisions such as maintaining the current lane, changing lanes, overtaking, turning left or right, steering, accelerating, decelerating, and stopping. Furthermore, the planning unit 30 generates a target trajectory TRJ necessary for vehicle 1 to drive according to the driving plan. The target trajectory TRJ includes a target position and a target speed.

[0021] The control quantity calculation unit 40 receives the target trajectory TRJ from the planning unit 30. The control quantity calculation unit 40 calculates the control quantity CON required for the vehicle 1 to follow the target trajectory TRJ. The control quantity CON can also be described as the control quantity required to reduce the deviation between the vehicle 1 and the target trajectory TRJ. The control quantity CON includes at least one of the steering control quantity, drive control quantity, and braking control quantity.

[0022] The running gear 50 includes a steering gear 51, a drive gear 52, and a braking gear 53. The steering gear 51 steers the wheels. The drive gear 52 is a power source that generates driving force. Examples of the drive gear 52 include an engine, an electric motor, etc. The braking gear 53 generates braking force. The running gear 50 receives a control amount CON from the control amount calculation unit 40. The running gear 50 operates the steering gear 51, the drive gear 52, and the braking gear 53 according to the steering control amount, the drive control amount, and the braking control amount, respectively. As a result, the vehicle 1 travels in accordance with the target trajectory TRJ.

[0023] The recognition unit 20 includes at least one of a rule-based model and a machine learning model. The rule-based model performs recognition processing based on a predetermined set of rules. Examples of machine learning models include NN (Neural Network), SVM (Support Vector Machine), regression models, decision tree models, etc. The NN may be a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or a combination thereof. The type of each layer, the number of layers, and the number of nodes in the NN are arbitrary. The machine learning model is generated in advance through machine learning. The recognition unit 20 performs recognition processing by inputting sensor detection information SEN into the model.

[0024] Similarly, the planning unit 30 includes at least one of a rule-based model and a machine learning model. The planning unit 30 performs planning by inputting recognition result information RES into the model.

[0025] Similarly, the control variable calculation unit 40 includes at least one of a rule-based model and a machine learning model. The control variable calculation unit 40 performs the control variable calculation process by inputting a target trajectory TRJ into the model.

[0026] Two or more of the recognition unit 20, planning unit 30, and control variable calculation unit 40 may be configured as a single unit. The recognition unit 20, planning unit 30, and control variable calculation unit 40 may all be configured as a single unit (end-to-end configuration). For example, the recognition unit 20 and planning unit 30 may be configured as a single unit by a neural network (NN) that outputs a target trajectory TRJ from sensor detection information SEN. Even in the case of a single unit configuration, intermediate products such as recognition result information RES and target trajectory TRJ may be output. For example, if the recognition unit 20 and planning unit 30 are configured as a single unit by an NN, the recognition result information RES may be the output of the intermediate layer of the NN.

[0027] The recognition unit 20, the planning unit 30, and the control variable calculation unit 40 constitute an "automatic driving control unit" that controls the automatic driving of the vehicle 1. In this embodiment, at least one of the recognition unit 20 or the planning unit 30 includes a machine learning model. That is, the machine learning model performs at least one of the following: recognition of the surrounding environment of the vehicle 1 or generation of a driving plan. When the machine learning model recognizes the surrounding environment of the vehicle 1, the inference result by the machine learning model is, for example, an object recognized on an image, or an object recognized on a spatial map of the surrounding environment of the vehicle 1. When the machine learning model generates a driving plan, the inference result by the machine learning model is, for example, a driving decision that constitutes the driving plan. The automatic driving control unit uses the machine learning model to perform automatic driving control of the vehicle 1.

[0028] Figure 2 shows an example configuration of the target control device 200. The control device 200 is mounted on the vehicle 1 and performs automatic driving control of the vehicle 1. The control device 200 has at least the functions of the automatic driving control unit described above. The control device 200 may be configured to communicate with the sensor group 10 and the driving device 50.

[0029] The control device 200 includes one or more processors 210 (hereinafter simply referred to as processor 210) and one or more storage devices 220 (hereinafter simply referred to as storage devices 220). The processors 210 execute various processes according to a computer program. The computer program may be stored in the storage devices 220. The storage devices 220 store various information. The recognition unit 20, the planning unit 30, and the control variable calculation unit 40 may be implemented by a single processor 210 or by separate processors 210.

[0030] Model data 221 is model data included in the recognition unit 20, the planning unit 30, and the control variable calculation unit 40. As described above, in this embodiment, at least one of the recognition unit 20 or the planning unit 30 includes a machine learning model. Model data 221 is stored in the storage device 220 and used for automatic driving control.

[0031] The control device 200 is configured to be able to be updated for purposes such as improving control performance or adding functions. For example, the control device 200 is configured to acquire an update computer program via a communication network and install the acquired computer program. In particular, the update of the control device 200 may include upgrading the machine learning model. In this case, the control device 200 acquires a new version of the machine learning model and replaces the machine learning model included in the model data 221 with the acquired new version of the machine learning model.

[0032] While performing autonomous driving control, the processor 210 collects log data LOG related to autonomous driving control using a machine learning model. The processor 210 stores the log data LOG collected during autonomous driving control in the storage device 220. The log data LOG may be a log of data changing over time, or a log of data relating to the position of vehicle 1.

[0033] The management server 102 is a database server that manages the database 103. The management server 102 can also be thought of as a storage device that stores the database 103. The management server 102 communicates with one or more vehicles 1 via a communication network. In this embodiment, during or after the execution of automated driving control, the processor 210 of vehicle 1 uploads a portion of the log data LOG stored in the local storage device 220 to the management server 102. In particular, the processor 210 is configured to upload log data LOG related to at least the inference results from the machine learning model to the management server 102. The log data LOG related to the inference results from the machine learning model may include version information of the machine learning model at the time the log data LOG was collected.

[0034] The management server 102 acquires log data LOG uploaded from one or more vehicles 1. The management server 102 then manages the acquired log data LOG in the database 103. The database 103 manages log data LOG related to the inference results of machine learning models for at least each vehicle. For example, the database 103 manages the log data LOG in association with the vehicle ID.

[0035] 2. Update Verification System The update verification method according to this embodiment is implemented by a process executed by a computer. Figure 3 shows an example of the hardware configuration of the update verification system 100 that implements the update verification method according to this embodiment.

[0036] The update verification system 100 includes a processing unit 101 and a management server 102.

[0037] The processing unit 101 is a computer that includes one or more processors 110 (hereinafter simply referred to as processor 110) and one or more storage devices 120 (hereinafter simply referred to as storage devices 120). The processing unit 101 is configured to communicate with the management server 102 and the control device 200. In particular, the processors 110 are configured to communicate with the management server 102 and access the database 103. The processing unit 101 may be a computer installed in the vehicle 1, or it may be an external computer configured to communicate with the vehicle 1.

[0038] The processor 110 performs various processes. In particular, the processor 110 performs a process to verify the update of the control device 200 (hereinafter referred to as the "update verification process"). Details of the update verification process will be described later. The processor 110 can be composed of, for example, a CPU (Central Processing Unit) including an arithmetic unit and registers. The storage device 120 stores various information necessary for the execution of processes by the processor 110. The storage device 120 can be composed of, for example, a recording medium such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive).

[0039] The storage device 120 stores the computer program 121 and verification data 122.

[0040] The computer program 121 is executed by the processor 110. Various processes by the processing unit 101 may be realized through the cooperation of the processor 110 executing the computer program 121 and the storage device 120. The computer program 121 may be recorded on a computer-readable recording medium.

[0041] Verification data 122 is data used in the update verification process. As will be described later, examples of verification data 122 include log data LOG obtained from the control device 200, log data LOG obtained from the management server 102, etc.

[0042] 3. Update Verification Method The update verification method implemented by the update verification system 100 described above will be explained below. Figure 4 is a diagram illustrating the update verification method according to this embodiment.

[0043] In Figure 4, the control unit 200 is first updated (S210). Here, the update of the control unit 200 includes upgrading the machine learning model.

[0044] After the update is performed, the control unit 200 collects log data LOG (S220). The collected log data LOG includes the inference results from the machine learning model after the version upgrade.

[0045] Next, the control device 200 sends a verification request to the processing unit 101 requesting verification of the update (S230). The verification request may include the ID information of vehicle 1 and the version information of the machine learning model before and after the update. Furthermore, the control device 200 may be configured to send log data LOG collected over a predetermined period or distance after the update has been performed, along with the verification request.

[0046] When the processor 110 of the processing unit 101 receives a verification request from the control device 200, it verifies the requested verification content (S110). For example, the processor 110 verifies the ID of the vehicle 1 on which the control device 200 to be verified is installed, verifies the version of the machine learning model before and after the update, and so on.

[0047] Next, the processor 110 requests log data from the management server 102 according to the verified results (S120).

[0048] In particular, in step S120, the processor 110 requests log data (hereinafter also referred to as "first log data") relating to the inference results of the updated machine learning model collected in the control device 200 under verification over a predetermined period (or distance). For example, the processor 110 requests log data specifying the ID of the vehicle 1 in which the control device 200 under verification is installed, the version of the updated machine learning model, and the period (or distance). However, the processor 110 may also be configured to obtain the first log data directly from the control device 200 under verification.

[0049] Furthermore, in step S120, the processor 110 requests log data (hereinafter also referred to as "second log data") relating to the inference results of the machine learning model in the version prior to the update over a predetermined period (or distance). For example, the processor 110 requests log data specifying the version of the machine learning model prior to the update and the period (or distance). The second log data does not have to be log data collected in the control device 200 under verification. In other words, the second log data may be log data uploaded to the management server 102 from another vehicle 1. The processor 110 may also be configured to request multiple sets of second log data by specifying multiple periods (or distances) or by specifying multiple other vehicles 1.

[0050] When the management server 102 receives a request for log data from the processing unit 101, it checks the request (S121) and sends the corresponding log data to the processing unit 101 (S122).

[0051] The processor 110 of the processing unit 101 acquires log data from the management server 102 (S130). The processor 110 stores the acquired first log data and second log data in the storage device 120 as verification data 122. Then, the processor 110 performs update verification processing based on the first log data and second log data (S140).

[0052] Figure 5 is a flowchart showing the processes executed by the processor 110 during the update verification process.

[0053] In step S141, the processor 110 obtains the first and second verification counts from the first and second log data, respectively. If the machine learning model recognizes the surroundings of vehicle 1, the first verification count is the number of objects around vehicle 1 recognized by the machine learning model in the first log data (for example, the number of bounding boxes surrounding the objects recognized in the image). The second verification count is the number of objects around vehicle 1 recognized by the machine learning model in the second log data. If the machine learning model generates a driving plan, the first and second verification counts may also be the number of driving decisions in the driving plan generated by the machine learning model in the first and second log data, respectively. If multiple second log data sets are obtained, the processor 110 may obtain the second verification count for each of the second log data sets.

[0054] Next, in step S142, the processor 110 compares the acquired first verification number with the second verification number.

[0055] The number of objects around vehicle 1 recognized by the machine learning model and the number of driving decisions in the generated driving plan are indicators of the performance of the machine learning model's inference results. In particular, under normal circumstances, these numbers are not expected to change drastically before and after the update. Considering that the previous version of the machine learning model would have already been in operation for a considerable time, if these numbers change significantly before and after the update, there is a possibility that the upgraded machine learning model is not functioning properly.

[0056] For example, if the first and second verification counts each represent the number of objects around vehicle 1 that were recognized, a significant increase in the first verification count compared to the second verification count may indicate a rapid increase in false detections. Conversely, a significant decrease in the first verification count compared to the second verification count may indicate a rapid increase in undetected objects. Similarly, if the first and second verification counts each represent the number of driving decisions in the generated driving plan, a significant change in the first verification count relative to the second verification count may indicate a rapid increase in false judgments or undecided objects.

[0057] In this way, by comparing the first verification count and the second verification count, it is possible to verify whether or not the update is successful. In particular, the first and second verification counts can be obtained from log data collected while the vehicle 1 is performing automatic driving control. That is, according to this embodiment, the update of the control device 200 can be verified without requiring special or additional verification. This makes it possible to identify early on if an update has been performed that is not successful.

[0058] If the first verification count has changed drastically compared to the second verification count (S143; Yes), the processor 110 determines that the update is not normal (S144) and terminates the process. On the other hand, if the first verification count has not changed drastically compared to the second verification count (S143; No), the processor determines that the update is normal (S145) and terminates the process.

[0059] As explained above, the processor 110 executes the update verification process. In the update verification process, the comparison between the first verification count and the second verification count can be performed, for example, as follows.

[0060] One method is to examine the magnitude of the difference between the number of first verification numbers per unit time (or unit distance) and the number of second verification numbers per unit time (or unit distance). Figure 6(A) is a conceptual diagram showing an example of comparing the first and second verification numbers by examining the magnitude of the difference. In Figure 6(A), the number of first verification numbers per unit time (or unit distance) and the number of second verification numbers per unit time (or unit distance) are shown by dashed lines. In this case, the processor 110 can determine that the update is not normal when the magnitude of the difference exceeds a predetermined threshold.

[0061] Another approach is to compare the statistical data obtained from multiple second verification numbers when multiple second verification numbers are acquired. Figure 6(B) is a conceptual diagram showing an example of comparing the first and second verification numbers based on statistical data obtained from multiple second verification numbers. In this case, the processor 110 calculates a statistical model based on the multiple second verification numbers regarding the number of objects around vehicle 1 recognized by the machine learning model and the number of driving decisions in the generated driving plan. If it determines that there is a significant difference between the first verification number and the calculated statistical model, it can determine that the update is not normal.

[0062] Refer to Figure 4 again. After the processor 110 of the processing unit 101 executes the update verification process, it sends the verification result to the control device 200 (S150). The control device 200 receives the verification result sent from the processing unit 101 (S160).

[0063] The control device 200 may be configured to perform processing in response to the received verification results. For example, the control device 200 may be configured to roll back to the state before the update when it receives a verification result indicating that the update was not successful.

[0064] As described above, the update verification system and update verification method according to this embodiment are realized. Furthermore, as mentioned above, the update verification program according to this embodiment is realized by the computer program 121 that causes the processor 110 to execute processing. [Explanation of symbols]

[0065] 1 vehicle, 100 update verification systems, 110 processors, 120 Storage device, 121 Computer program, 200 Control device

Claims

1. A method for verifying updates to a control system that performs automated driving control of a vehicle using a machine learning model, using a computer to verify the update, The computer is configured to access one or more storage devices that store log data relating to the inference results of the machine learning model. The aforementioned update includes upgrading the machine learning model, The machine learning model performs at least one of the following: recognition of the surrounding environment of the vehicle or generation of a driving plan for automated driving control. The aforementioned update verification method is: The upgraded machine learning model obtains a first verification number indicating the number of objects around the vehicle recognized over a predetermined period or distance, or the number of driving decisions in the generated driving plan; Referencing the log data, obtain a second verification number indicating the number of objects around the vehicle recognized by the machine learning model in the pre-update version during a predetermined period or distance, or the number of driving decisions in the generated driving plan; The first verification count and the second verification count are compared to verify whether the update is successful or not, This includes performing the above on the computer. How to verify the update.

2. The update verification method according to claim 1, Verifying whether the update is successful includes determining that the update is not successful when the magnitude of the difference between the number of first verifications per unit time or unit distance and the number of second verifications per unit time or unit distance exceeds a predetermined threshold. Characterized by How to verify the update.

3. The update verification method according to Claim 1, This includes obtaining multiple of the above-mentioned second verification numbers, Verifying whether the aforementioned update is successful is necessary. Based on a plurality of the aforementioned second verification numbers, a statistical model is calculated relating to the number of objects around the vehicle recognized by the machine learning model or the number of driving decisions in the generated driving plan, If there is a significant difference in the first validation number compared to the aforementioned statistical model, the update is deemed not to be normal. including How to verify the update.

4. The update verification method according to Claim 3, The multiple second verification numbers are the second verification numbers obtained from the log data collected from multiple vehicles. How to verify the update.

5. An update verification system for verifying updates to control devices that perform automated driving control of vehicles using machine learning models, One or more processors, One or more storage devices for storing log data relating to the inference results of the machine learning model, Includes, The aforementioned update includes upgrading the machine learning model, The machine learning model performs at least one of the following: recognition of the surrounding environment of the vehicle or generation of a driving plan for automated driving control. The one or more processors described above are: The process involves obtaining a first verification number, which indicates the number of objects around the vehicle recognized over a predetermined period or distance by the upgraded machine learning model, or the number of driving decisions in the generated driving plan. The process involves referring to the log data to obtain a second verification number indicating the number of objects around the vehicle recognized by the machine learning model in the pre-update version during a predetermined period or distance, or the number of driving decisions in the generated driving plan; A process to verify whether the update is successful by comparing the first verification count and the second verification count, It is configured to execute Update verification system.

6. An update verification program that uses a machine learning model to have a computer perform verification of an update to a control device that performs autonomous driving control of a vehicle, The computer is configured to access one or more storage devices that store log data relating to the inference results of the machine learning model. The aforementioned update includes upgrading the machine learning model, The machine learning model performs at least one of the following: recognition of the surrounding environment of the vehicle or generation of a driving plan for automated driving control. The aforementioned update verification program, The process involves obtaining a first verification number, which indicates the number of objects around the vehicle recognized over a predetermined period or distance by the upgraded machine learning model, or the number of driving decisions in the generated driving plan. The process involves referring to the log data to obtain a second verification number indicating the number of objects around the vehicle recognized by the machine learning model in the pre-update version during a predetermined period or distance, or the number of driving decisions in the generated driving plan; A process to verify whether the update is successful by comparing the first verification count and the second verification count, The system is configured to cause the computer to execute the above. Update verification program.

Images