On-board arithmetic processing device and arithmetic processing method

Abstract

To switch to an appropriate AI model by verifying whether degradation in accuracy is caused due to switching the AI model.SOLUTION: An in-vehicle computational processing device that performs computations using a trained model includes: a computational unit that performs computations using a first trained model;, a state estimation unit that estimates a state by acquiring a position of a vehicle or information acquired from the inside and outside the vehicle; and a model selection unit that selects a second trained model on the basis of the state; an a model switching verification unit that compares a computational result of the first trained model with a computational result obtained by performing computations using the second trained model in parallel with the first trained model, and determines, on the basis of a result of the comparison, whether to switch from the first trained model to the second trained model. If the model switching verification unit has determined to switch to the second trained model, the computational unit switches from the first trained model to the second trained model to perform computations.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an on-board arithmetic processing device and an arithmetic processing method that recognize the external environment of a vehicle by processing the output of an external sensor mounted on the vehicle using an AI model that has undergone machine learning. [Background technology]

[0002] In recent years, vehicles equipped with autonomous driving (AD) systems and advanced driver-assistance systems (ADAS) are becoming more common. To realize these systems, it is necessary to recognize the external environment of the vehicle (other vehicles, pedestrians, bicycles, road surfaces, white lines, road signs, obstacles, and their positions, etc.). For this reason, a method is often adopted to recognize the external environment of the vehicle by processing the output of external sensors mounted on the vehicle with a machine-learned AI model (hereinafter simply referred to as "AI model" or "model").

[0003] Here, if one AI model is used to recognize the external environment for all scenes while the vehicle is traveling, it is necessary to prepare an AI model that has been machine-learned using comprehensive sensor output data in advance. However, in order to ensure recognition accuracy in a variety of scenes, such an AI model requires an exponential increase in model size, and the ECU (Electronic Control Unit) that runs the AI ​​model also requires extremely high computing performance. As a result, there are problems in that the chip cost of the ECU increases, and the computing load (power consumption) during autonomous driving also increases.

[0004] On the other hand, if a small AI model trained on limited data is applied in order to reduce ECU chip costs and computational load (power consumption), there is a high possibility that the accuracy of recognizing the external environment will be significantly degraded in environments that the AI ​​model cannot handle.

[0005] In consideration of these issues, a method is being considered in which multiple relatively small AI models are prepared in advance and then appropriately switched to the AI ​​model appropriate for the driving scene or driving position, thereby improving the accuracy of recognizing the external environment in any driving scene while also reducing the calculation load (power consumption) of the ECU.

[0006] For example, the abstract of Patent Document 1 states that the problem is "When a trained model for controlling a vehicle is set for each specified area, the trained model is appropriately switched depending on the vehicle's position." and states as a solution, "A control assistance device that assists in vehicle control using a trained model based on machine learning, the control device including a control unit that can be updated in the vehicle based on vehicle information and position information of the vehicle transmitted from the vehicle, and that selects a trained model that corresponds to the vehicle's current position included in the position information, and a transmission unit that transmits the selected trained model to the vehicle." [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2020-93760 A Summary of the Invention [Problem to be solved by the invention]

[0008] As is clear from the description in the abstract, such as "a control unit that selects a trained model that corresponds to the vehicle's current position," the control assistance device of Patent Document 1 switches the AI ​​model to be applied in response to changes in the vehicle's position.

[0009] However, Patent Document 1 does not verify whether the recognition accuracy deteriorates when switching the AI ​​model. If an inappropriate AI model is switched to, the recognition accuracy of the external environment may deteriorate, which may result in destabilization of vehicle control by AD or ADAS.

[0010] The present invention has been made in consideration of the above problems, and aims to provide an on-board arithmetic processing device and an arithmetic processing method that, when switching AI models, selects an AI model that does not degrade the accuracy of recognizing the external environment. [Means for solving the problem]

[0011] In order to solve the above problems, the in-vehicle arithmetic processing device of the present invention is mounted on a vehicle and performs a calculation using a learned model, and includes a calculation unit that performs a calculation using a first learned model, a state estimation unit that estimates the position of the vehicle or an acquired state inside and outside the vehicle, and a model selection unit that selects a second learned model based on the state, and a model switching verification unit that compares a calculation result when the calculation is performed in parallel with the first learned model using the second learned model with a calculation result of the first learned model and decides whether to switch from the first learned model to the second learned model based on the comparison result, and when it is decided by the model switching verification unit to switch to the second learned model, the calculation unit is an in-vehicle arithmetic processing device that switches from the first learned model to the second learned model and performs the calculation. Effect of the Invention

[0012] According to the in-vehicle arithmetic processing device and arithmetic processing method of the present invention, the accuracy of recognizing the external environment does not deteriorate even when the AI ​​model is switched. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a functional block diagram of the in-vehicle arithmetic processing device according to the first embodiment. [Diagram 2] 4 is a flowchart showing a processing procedure of the in-vehicle arithmetic processing device according to the first embodiment. [Figure 3A] FIG. 4 is a diagram showing an example of a driving scene state table, which is a type of state table. [Figure 3B] FIG. 13 is a diagram showing an example of a user status table, which is a type of status table. [Figure 3C]FIG. 13 is a diagram showing an example of an apparatus load status table, which is a type of status table. [Figure 4] FIG. 13 is a diagram showing an example of a model table. [Figure 5A] Functional block diagram of model validation using Lider output as the correct answer. [Figure 5B] FIG. 1 is a functional block diagram of model verification in which outputs from different types of sensors are used as correct values. [Figure 6A] Functional block diagram of model verification in which the output of the current model is the correct answer. [Figure 6B] Functional block diagram of model verification in which the output of the current model is the correct answer. [Figure 7] FIG. 4 is a diagram showing a display panel during autonomous driving according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing a setting screen during automatic driving according to the first embodiment. [Figure 9] FIG. 11 is a functional block diagram of an in-vehicle arithmetic processing device according to a second embodiment. [Figure 10] 10 is a flowchart showing a processing procedure of an in-vehicle arithmetic processing device according to the second embodiment. [Figure 11] FIG. 13 is a diagram showing an example of a model difference table. [Figure 12] FIG. 11 is a diagram showing an example of a method for displaying a past driving history according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES

[0014] First, an on-board arithmetic processing device 1 according to a first embodiment of the present invention will be described with reference to Fig. 1 to Fig. 8. In the following description, the on-board arithmetic processing device 1 is described as being built in an ECU, but the on-board arithmetic processing device 1 may be built in an external sensor, and the external environment recognized by the external sensor may be output to the ECU.

[0015] Fig. 1 is a functional block diagram of an in-vehicle arithmetic processing device 1 according to Example 1. As shown in Fig. 1, the in-vehicle arithmetic processing device 1 has a main arithmetic unit 11, an AI arithmetic unit 12, and a storage unit 13. Note that, although Fig. 1 illustrates a structure in which the main arithmetic unit 11 and the AI ​​arithmetic unit 12 are separated, both arithmetic units may be integrated.

[0016] As shown in the figure, the in-vehicle arithmetic processing device 1 is a device that receives external sensor information I1 from an external sensor and vehicle sensor information I2 from a vehicle sensor. Here, the external sensor is a sensor that acquires information about the surroundings of the vehicle, such as a stereo camera, a monocular camera, a LiDAR, an illuminance sensor, an acceleration sensor, a radar, etc. In addition, the vehicle sensor is a sensor that acquires information related to the position and state of the vehicle, such as a car navigation system, a GNSS (Global Navigation Satellite System) receiver, a speed sensor, a steering angle sensor, a fuel gauge, a battery gauge, etc. Note that the external sensor information I1 and the vehicle sensor information I2 do not necessarily have to be input one by one, and a plurality of external sensor information I1 and vehicle sensor information I2 may be input.

[0017] The main calculation unit 11 is a unit that executes a predetermined program by a calculation device such as a CPU (Central Processing Unit) or an FPGA (Field Programmable Gate Array). By executing the predetermined program, the main calculation unit 11 realizes functional units such as a state estimation unit 11a, a candidate model selection unit 11b, a current model control unit 11c, a candidate model control unit 11d, a result receiving unit 11e, and a model verification unit 11f. The AI ​​calculation unit 12 is an AI accelerator specialized in processing the AI ​​model, and has a model calculation unit 12a and an AI calculation control unit 12b. The memory unit 13 is a storage device such as a semiconductor memory, and has a state table 13a, a model table 13b, and a parameter table 13c. Below, the details of each unit of the in-vehicle calculation processing device 1 will be described mainly along the flow of information.

[0018] <Details of each part of the on-board processing unit 1> The state estimation unit 11a of the main calculation unit 11 estimates the current state of the vehicle based on the input external sensor information I1 and vehicle sensor information I2. The state estimated here may be, for example, the state of the external environment of the vehicle, which is a combination of the type of the vehicle's driving position (highway, general road, parking lot, etc.), the type of current time (morning, noon, evening, night), and the type of weather (sunny, cloudy, rain, snow, etc.). The state estimated by the state estimation unit 11a may be the resource state of the in-vehicle calculation processing device 1, such as the level of power consumption in the vehicle, the level of battery remaining, and the number of recognized objects, or who is the driver. The state estimation by the state estimation unit 11a is assumed to be a rule-based estimation using environmental information from a single or multiple external sensor information I1, or an estimation using an AI model. The road conditions during driving may also be estimated from a map and position information as the vehicle sensor information I2.

[0019] When the state estimated by the state estimation unit 11a changes, the candidate model selection unit 11b of the main calculation unit 11 refers to the state table 13a of the storage unit 13 and selects the model ID of the AI ​​model linked to the estimated state. Details of the state table 13a will be described later.

[0020] The candidate model control unit 11d of the main calculation unit 11 refers to the model table 13b of the storage unit 13 to acquire model information corresponding to the model ID selected by the candidate model selection unit 11b, and also refers to the parameter table 13c to acquire model parameters corresponding to the selected model ID. Then, the candidate model control unit 11d transmits the acquired model information and model parameters to the AI ​​calculation unit 12. Furthermore, the candidate model control unit 11d stops the environment recognition process using the candidate model M2 in the AI ​​calculation unit 12 as necessary. Details of the model table 13b will be described later.

[0021] The AI ​​calculation control unit 12b of the AI ​​calculation unit 12 transmits the model information and model parameters received from the candidate model control unit 11d to the model calculation unit 12a. In addition, the AI ​​calculation control unit 12b starts or stops the environment recognition process in the model calculation unit 12a based on a command from the main calculation unit 11.

[0022] The model calculation unit 12a of the AI ​​calculation unit 12 has a structure specialized for vector calculations and tensor calculations that are frequently used in environmental recognition processing using an AI model, and can perform environmental recognition processing using the current model M1 and environmental recognition processing using the candidate model M2 in parallel. The current model M1 (first trained model) is an AI model that is currently used for AD or ADAS and was confirmed to have a predetermined level of accuracy or higher in the previous verification. The candidate model M2 (second trained model) is an AI model that may be used in the future instead of the current model M1 and is constructed using model information and model parameters received from the candidate model control unit 11d.

[0023] If necessary, the current status model control unit 11c of the main calculation unit 11 stops the environment recognition process using the current status model M1 in the AI ​​calculation unit 12. In addition, if necessary, the current status model control unit 11c sets the candidate model M2 as a new current status model M1.

[0024] The result receiving unit 11e of the main calculation unit 11 receives the recognition results of the current model M1 and the candidate model M2 from the AI ​​calculation unit 12, and transmits them to the model verification unit 11f.

[0025] The model verification unit 11f of the main calculation unit 11 verifies the accuracy of both models by comparing the two recognition results received from the result receiving unit 11e, and judges whether the AI ​​model used in AD or ADAS can be switched from the current model M1 to the candidate model M2. If the model verification unit 11f judges that it is possible to switch to the candidate model M2, it sends a stop signal to the current model control unit 11c, and if it judges that it is not possible to switch to the model M2, it sends a stop signal to the candidate model control unit 11d. As a result, the current model control unit 11c or the candidate model control unit 11d can stop one of the AI ​​models being processed by the model calculation unit 12a of the AI ​​calculation unit 12 according to the judgment result of the model verification unit 11f.

[0026] <Processing Procedure of the In-Vehicle Processing Device 1> Next, the processing procedure of the in-vehicle processing device 1 will be described using the flowchart in Fig. 2. In Fig. 2, the processing of the main processing unit 11 and the processing of the AI ​​processing unit 12 are shown separately, and the arrows that cross both processing flows indicate the transmission of data.

[0027] At the time of step S21, the model calculation unit 12a of the AI ​​calculation unit 12 is executing only the previous current model M1 as the AI ​​model used in AD and ADAS.

[0028] In step S11, the state estimation unit 11a of the main calculation unit 11 acquires the external sensor information I1 and the vehicle sensor information I2, and estimates the current vehicle state from the environmental information. Note that the vehicle state estimated here is, as described above, for example, the type of the vehicle's running position (highway, general road, parking lot, etc.), the type of current time (morning, noon, evening, night), the type of weather (sunny, cloudy, rain, snow, etc.), the level of power consumption in the vehicle, the level of remaining battery power, the number of objects to be recognized, etc.

[0029] In step S12, the candidate model selection unit 11b of the main calculation unit 11 refers to the state table 13a stored in the storage unit 13, and selects a candidate model M2 corresponding to the state estimated by the state estimation unit 11a. Hereinafter, a method of selecting the candidate model M2 with reference to the state table 13a will be specifically described with reference to Figures 3A to 3C.

[0030] FIG. 3A is an example of the driving scene state table 13a1, which is a type of the state table 13a, and is a table for identifying a candidate model M2 according to the driving scene. For example, if the driving scene estimated by the state estimation unit 11a is a driving scene with a combination of a driving position type "general road", weather "sunny", and a current time "morning", the candidate model selection unit 11b selects the model ID "m01" as the candidate model M2. Note that the driving scenes registered in the driving scene state table 13a1 are not limited to those exemplified in FIG. 3A, and snow, mountain roads, tunnels, etc. may be registered as driving scenes. In addition, as a method of dividing the models, a model may be prepared for each combination of multiple states, such as the driving scenes shown in FIG. 3A, or one model may be created by combining morning, afternoon, and night states, such as general roads, expressways, and parking lots.

[0031] 3B is an example of a user state table 13a2, which is a type of the state table 13a, and is a table for identifying a candidate model M2 corresponding to a driver (user). In this table, a model ID is registered for each user, and when the state estimation unit 11a detects a change of user, such as when a user setting is changed, the candidate model selection unit 11b selects a model ID corresponding to the new user as the candidate model M2.

[0032] 3C is an example of a device load state table 13a3, which is a type of state table 13a, and is a table for identifying a candidate model M2 according to the load state of the in-vehicle calculation processing device 1. In this table, AI models are registered for each magnitude of device load, and, for example, if the current device load is "high load", model ID "m15" corresponding to a model with a relatively small calculation load is selected, and if the current device load is "low load", model ID "m16" corresponding to a model with a relatively large calculation load is selected.

[0033] In step S13, the candidate model control unit 11d of the main calculation unit 11 refers to the model table 13b and the parameter table 13c of the storage unit 13, and acquires model information and model parameters associated with the model ID selected by the candidate model selection unit 11b. Then, the candidate model control unit 11d transmits to the AI ​​calculation unit 12, in addition to the model information and model parameters, a processing start signal for the candidate model M2 generated based on the information.

[0034] FIG. 4 is a diagram showing an example of the model table 13b. The model table 13b shown here holds information on "model ID", "number of cores", "input size", "number of layers", and "weight size list for each layer". In the "model ID", a model ID that can be selected by referring to the state table 13a shown in FIGS. 3A to 3C is registered. In the "number of cores", the number of cores of the AI ​​calculation unit 12 required for executing the AI ​​model linked to the "model ID" is registered. In the "input size", the input size of the recognition target data to the AI ​​model linked to the "model ID" is registered. In the "number of layers", the number of layers of the AI ​​model linked to the "model ID" is registered. In the "weight size list for each layer", the weight size of each layer of the AI ​​model linked to the "model ID" is registered. Note that the information stored and acquired in the model table 13b is not limited to this, and necessary information can be held depending on the computer architecture of the AI ​​calculation unit 12.

[0035] In step S22, the model calculation unit 12a of the AI ​​calculation unit 12 additionally executes the candidate model M2 constructed using the model information and model parameters. Note that since the model calculation unit 12a has already executed the current model M1 at the time of step S21, both the current model M1 and the candidate model M2 are executed in this step. The execution results of both models are sent to the result receiving unit 11e of the main calculation unit 11.

[0036] In step S14, the result receiving unit 11e of the main calculation unit 11 acquires the execution results of the current model M1 and the candidate model M2.

[0037] In step S15, the model verification unit 11f of the main calculation unit 11 verifies the processing results of the current model M1 and the candidate model M2 of the output of the external sensor information I1, etc., and determines whether or not to switch to the candidate model M2 based on the verification result. The details of the verification method here will be described later.

[0038] Step S16 branches depending on the decision in step S15. That is, if it is decided that switching to the candidate model M2 is possible, the process proceeds to step S23, and if it is decided that switching is not possible, the process proceeds to step S17.

[0039] In step S23, the AI ​​calculation control unit 12b of the AI ​​calculation unit 12 stops the execution of the current model M1 in response to the command received from the current model control unit 11c of the main calculation unit 11, sets the candidate model M2 as the future current model M1, and ends the processing in Fig. 2. As a result, in the model calculation unit 12a, AD or ADAS using the new current model M1 is started.

[0040] In step S17, the model verification unit 11f of the main calculation unit 11 judges whether the previous current state model M1 is still valid in the current environment. If the recognition accuracy of the previous current state model M1 is equal to or higher than the threshold, the process proceeds to step S24; otherwise, the process proceeds to step S25.

[0041] In step S24, the AI ​​calculation control unit 12b of the AI ​​calculation unit 12 stops the execution of the candidate model M2 in response to the command received from the candidate model control unit 11d of the main calculation unit 11, and ends the processing in Fig. 2. As a result, in the model calculation unit 12a, only the previous current status model M1 is executed, and the AD and ADAS using the previous current status model M1 are continued.

[0042] In step S25, the AI ​​calculation control unit 12b of the AI ​​calculation unit 12 stops the execution of both the current model M1 and the candidate model M2 in response to commands received from the current model control unit 11c and the candidate model control unit 11d of the main calculation unit 11, and ends the processing in Fig. 2. As a result, the AD and ADAS are stopped and the driving mode is switched to a manual driving mode by the user, but the execution of inappropriate AD and ADAS due to a misperception of the external environment is prevented.

[0043] In the above step S12, one candidate model M2 is selected, but if multiple candidate models M2 can be identified by referring to the state table 13a, all of the multiple candidate models M2 may be selected. In that case, all the selected candidate models M2 may be implemented in step S22, and the most accurate candidate model M2 may be set as the new current model M1 in step S23.

[0044] <Verification method in step S15> Here, the method for verifying the accuracy of the AI ​​model in step S15 will be described in detail. The accuracy of the AI ​​model can be verified by comparing the correct answer value with the recognition result of the AI ​​model, and the larger the error, the lower the accuracy. However, since it is difficult to obtain a completely correct answer value of the AI ​​model within the vehicle, in this embodiment, a value considered to be close to the correct answer value is compared with the output value of the AI ​​model, and the accuracy of each AI model is calculated from the error between the two.

[0045] In the following, the accuracy calculation method in step S15 will be described using an example in which the current model M1 and the candidate model M2 are AI models that recognize image data captured by a camera. In addition, the accuracy calculation method by comparison with values ​​close to the correct answer value will be described using two examples.

[0046] In the driving scene state table 13a1 illustrated in Fig. 3, for example, when switching from an AI model for general roads to an AI model for expressways, the recognition target may be partially different before and after the switching. In this case, a method of calculating the model accuracy may be considered in which the accuracy of each AI model is calculated using a value obtained by a sensor other than the camera as the correct answer value. Such a verification method will be described with reference to Figs. 5A and 5B.

[0047] In addition, in the driving scene state table 13a1 illustrated in Fig. 3, for example, when switching from a sunny / cloudy AI model to a night AI model, the recognition target may be the same before and after the switching. In this case, a method of calculating the model accuracy may be considered in which the recognition result of the current model M1 is used as the correct answer value to calculate the accuracy of the candidate model M2. Such a verification method will be described with reference to Figs. 6A and 6B.

[0048] <<Verification method using output from a sensor other than the camera as the correct value>> 5A is a functional block diagram illustrating a method of calculating the accuracy of each AI model using the output of Lider3 as a correct answer value and verifying each AI model. In this case, the result receiving unit 11e receives the result of recognizing the image captured by the camera 2 using the current model M1 and the candidate model M2, and also receives the result of recognizing the output of Lider3 using the model for Lider. Then, the model verification unit 11f verifies the model accuracy of the current model M1 and the candidate model M2 using the recognition result based on the output of Lider3 as a correct answer value.

[0049] 5B is a functional block diagram illustrating a method for calculating the accuracy of each AI model by using a value obtained by combining the outputs of multiple heterogeneous sensors 4 other than the camera as a correct answer value, and performing AI model verification. In this case, the result receiving unit 11e receives the results of the current model M1 and the candidate model M2 recognizing the image captured by the camera 2, and also receives the surrounding recognition result obtained by combining the outputs of the multiple heterogeneous sensors 4 in the fusion unit 41. Then, the model verification unit 11f verifies the model accuracy of the current model M1 and the candidate model M2 by using the recognition result based on the outputs of the heterogeneous sensors 4 as a correct answer value.

[0050] As shown in FIGS. 5A and 5B, the model verification unit 11f of this embodiment includes an accuracy calculation unit 11f1 and an accuracy comparison unit 11f2.

[0051] The accuracy calculation unit 11f1 calculates the current model accuracy from the magnitude of error between the correct value (the recognition result based on the output of Lidar3, or the recognition result based on the output of the heterogeneous sensor 4) and the recognition result of the current model M1, and also calculates the candidate model accuracy from the magnitude of error between the correct value and the recognition result of the candidate model M2.

[0052] Then, the accuracy comparison unit 11f2 compares the current model accuracy with the candidate model accuracy, and when the candidate model accuracy is higher than the current model accuracy and reaches the reference value, it determines that it is possible to switch from the current model M1 to the candidate model M2. On the other hand, when the current model accuracy is higher than the candidate model accuracy, the accuracy comparison unit 11f2 determines that it is impossible to switch from the current model M1 to the candidate model M2. Note that, when the candidate model accuracy is higher than the current model accuracy but is not sufficient for the reference value, it is possible that the accuracy of the AI ​​model itself is degraded, so it is possible to disable both models and stop AD and ADAS.

[0053] <<Verification method using the output of the current model M1 as the correct value>> 6A is a functional block diagram showing an example of a method for calculating the accuracy of a candidate model M2 using the recognition result of a current model M1 as a correct answer value. In the figure, a result receiving unit 11e receives the results of recognition by the current model M1 and the candidate model M2 using an image from a camera 2 as an input. Then, a model verification unit 11f compares the recognition result of the current model M1 with the recognition result of the candidate model M2 in an accuracy comparison unit 11f2, using the recognition result of the current model M1 as a correct answer value.

[0054] Figure 6B is a functional block diagram showing another example of calculating the accuracy of the candidate model M2 using the recognition result of the current model M1 as the correct value. In the result receiving unit 11e of this figure, in addition to the inputs from the current model M1 and the candidate model M2, vehicle speed information I2a, which is a type of vehicle sensor information I2, is also input. Further, the result receiving unit 11e in Figure 6B is provided with a recognition prediction unit 11e1 that predicts the current recognition result after the state change from the recognition result of the current model M1 that was operating before the state change detected by the state estimation unit 11a and the vehicle speed information I2a. In the accuracy comparison unit 11f2, the value predicted by the recognition prediction unit 11e1 is used as the correct value to compare the correct value with the recognition result of the candidate model M2.

[0055] <Notification Method for AI Model Switching> Figure 7 is a diagram showing an example of the display panel 5 on the car navigation etc. after the AI model is switched in the in-vehicle arithmetic processing device 1. In addition to the map information 5a, the display panel 5 performs mark displays representing driving scenes such as the automatic driving mode display 5b and the night driving mode display 5c. This makes it possible to confirm whether the model switching has been correctly achieved by the model verification unit 11f.

[0056] Figure 8 is a diagram showing an example of the automatic driving setting screen 5d that can be displayed on the display panel 5. This figure shows the setting of whether automatic driving is possible for each driving scene and whether switching between scenes is possible. When, as in this embodiment, automatic driving on ordinary roads is possible and automatic driving on highways is possible, and switching between ordinary roads and highways is set to be impossible during driving, model switching during driving is impossible. When switching between ordinary roads and highways is set to be possible, it can be confirmed that model verification during driving is possible and model switching is possible.

[0057] According to the in-vehicle computing device of the present embodiment described above, even when the AI ​​model is switched, the recognition accuracy of the external environment does not deteriorate. Therefore, even when a small AI model is used while being switched, the recognition accuracy does not deteriorate before and after the model switching, so that it is possible to reduce the chip cost of the ECU and the power consumption while preventing erroneous control of the vehicle due to poor recognition of the external environment, compared to the case where a large AI model is used to handle various situations.

[0058] In addition, since model verification using the on-board computing device of this embodiment is performed within the vehicle, unlike model verification performed on an external server connected via a communication line, it is possible to avoid delays in the model verification process and situations in which model verification is not possible due to poor communication in remote areas. EXAMPLES

[0059] Second Embodiment Next, an on-board processor 1 according to a second embodiment of the present invention will be described with reference to Fig. 9 to Fig. 12. Note that, in the following, a duplicated description of points common to the first embodiment will be omitted.

[0060] In the model calculation unit 12a of the first embodiment, before and after model switching, recognition processing using the current model M1 and recognition processing using the candidate model M2 were performed in parallel. However, the channels constituting both models also had common channels, and there was a problem in that the calculation load (power consumption) increased by operating the common channels in an overlapping manner.

[0061] Therefore, in this embodiment, when the model calculation unit 12a operates the current model M1 and the candidate model M2 in parallel, it is possible to obtain both the recognition results from the current model M1 and the recognition results from the candidate model M2 without consuming processing resources for two models, and use them to determine whether or not to switch the AI ​​model.

[0062] Therefore, in this embodiment, when creating an AI model for each driving scene, one large model is used to create a model covering multiple driving scenes using all learning data, and the created model is compressed for each driving scene using the learning data included in each driving scene. In this way, it is possible to create models for each driving scene in a form in which some of the models are shared. In addition, by creating a large model that covers all driving scenes and compressing it so that the amount of calculation is different, multiple models can be generated in a form in which some of the models are shared. In this way, when the AI ​​models for each driving scene share parts, it is possible to obtain the recognition results required for verification by additionally operating only the difference between the current model M1 and the candidate model M2 when the candidate model M2 is operating.

[0063] This embodiment assumes an example of model switching in which the AI ​​models for each driving scene share parts of each other in this manner.

[0064] 9 is a functional block diagram of the in-vehicle processing device 1 of this embodiment. The in-vehicle processing device 1 of this embodiment includes a main calculation unit 11, an AI calculation unit 12, and a storage unit 13, similar to the in-vehicle processing device 1 of the first embodiment (see FIG. 1). However, as a configuration unique to this embodiment, a model difference extraction unit 11g is added to the main calculation unit 11, and a model difference table 13d is added to the storage unit 13.

[0065] The model difference extraction unit 11g refers to the model table 13b (see FIG. 4) and the model difference table 13d based on the model ID selected by the candidate model selection unit 11b, and extracts difference channels between the candidate model M2 and the current model M1.

[0066] The candidate model control section 11d inputs the extracted channel difference information to the AI ​​calculation control section 12b of the AI ​​calculation section 12, and the model calculation section 12a loads and executes only the candidate model difference D2 as a model parameter.

[0067] When the model verification unit 11f can switch to the candidate model M2 based on the execution result, it stops the current model difference D1 while continuing the execution of the shared model M3. This can provide the same effect as stopping the current model M1. Note that in this embodiment, the model difference is assumed to be a channel, but the difference is not limited to this and may be a model layer unit or a model neuron unit.

[0068] Fig. 10 is a diagram showing the processing flow of the in-vehicle processing device 1 shown in Fig. 9, and shows the processing timing of the main processing unit 11 and the AI ​​processing unit 12. The arrows indicate data transmission. Note that, in the following, overlapping explanations of points common to Fig. 2 will be omitted.

[0069] At the time of step S21, the model calculation unit 12a of the AI ​​calculation unit 12 executes only the previous current model M1 as the AI ​​model used in AD and ADAS. Note that the current model M1 and the candidate model M2 themselves are not registered in the model calculation unit 12a of this embodiment, but are registered in the form of the shared model M3, the current model difference D1, and the candidate model difference D2. Therefore, in this embodiment, the execution of the current model M1 means the execution of both the shared model M3 and the current model difference D1, and the execution of the candidate model M2 means the execution of both the shared model M3 and the candidate model difference D2.

[0070] Steps S11 and S12 are the same as those in FIG.

[0071] In step S12a, the model difference extraction unit 11g of the main calculation unit 11 refers to the model table 13b and the model difference table 13d in the storage unit 13, and extracts the model difference between the current model M1 and the selected candidate model M2.

[0072] In step S13a, the candidate model control unit 11d of the main calculation unit 11 transmits to the AI ​​calculation unit 12 a signal to start processing using the candidate model M2, the model difference information, and the model difference parameters.

[0073] In step S22a, the model calculation unit 12a of the AI ​​calculation unit 12 additionally executes the candidate model difference D2. As described above, since the model calculation unit 12a has already executed both the shared model M3 corresponding to the current model M1 and the current model difference D1 at the time of step S21, by additionally executing the candidate model difference D2 in this step, it is possible to obtain an output equivalent to that obtained by executing both the current model M1 and the candidate model M2. The execution results of both models are transmitted to the main calculation unit 11.

[0074] Steps S14 to S17 and steps S23 to S25 are basically the same as those in Fig. 2. However, in this embodiment, stopping the current model M1 in step S23 corresponds to stopping only the current model difference D1, and stopping the candidate model M2 in step S24 corresponds to stopping only the candidate model difference D2.

[0075] FIG. 11 is an example of a diagram showing an example of the model difference table 13d in the storage unit 13 of the in-vehicle arithmetic processing device 1 shown in FIG. 9. As shown here, the model difference table 13d holds a model ID and an unnecessary channel list for each layer. In this embodiment, the unnecessary channel list in the model difference table 13d expresses the difference by how many fewer channels there are in each layer of the other models compared to the model m01. For example, the model m02 indicates that the 4th, 8th, 9th, and 30th channels are unnecessary in the first layer L1 compared to the model m01, and the 1st and 2nd channels are unnecessary in the second layer L2. Conversely, locations with more channels than the model m01 can be expressed in minus notation. EXAMPLES

[0076] Third Embodiment Next, an on-board processor 1 according to a third embodiment of the present invention will be described with reference to Fig. 12. In the following, overlapping descriptions of points common to the above-mentioned embodiments will be omitted.

[0077] In this embodiment, an example will be described in which the verification by the model verification unit 11f in the on-board arithmetic processing device 1 shown in the embodiments 1 and 2 is omitted based on past driving history. Since the more frequent the model verification in the on-board arithmetic processing device 1 is, the greater the processing load and the mounting size on the chip will be, so that in order to reduce the frequency, it is possible to reduce the number of verifications by switching to a model associated with similar timing and similar scenes based on past driving history such as past driving routes and driving environments.

[0078] FIG. 12 shows an example of a past driving history displayed on the display panel 5 of a vehicle. As shown here, the past driving history is displayed with the time, route name, starting point, destination, and user ID. By selecting a past driving history from this display, the user can switch to a similar model at the same timing as the previous driving. Note that the past driving history is not limited to that of a single vehicle, and it may be possible to search for the driving history of another vehicle obtained from a server or the like. [Explanation of symbols]

[0079] 1. On-board computing device 11 Main calculation section 11a State Estimation Unit 11b Candidate model selection section 11c Current model control section 11d Candidate model control section 11e Result Reception Unit 11f Model Verification Department 11f1 Accuracy calculation section 11f2 Accuracy comparison section 11g Model Difference Extraction Unit 12 AI calculation section 12a Model calculation section M1 current model M2 candidate model M3 Shared Model D1 Current model difference D2 Candidate model difference 12b AI calculation control unit 13 Storage section 13a Status Table 13b Model table 13c Parameter Table 13d model difference table 2 Camera 3. Lider 4. Heterogeneous Sensors 41 Fusion Club 5 Display Panel I1 External sensor information I2 Vehicle sensor information I2a Vehicle speed information

Claims

1. An on-board arithmetic processing device that is mounted on a vehicle and performs calculations using a trained model, A calculation unit that performs calculations using the first trained model; A state estimation unit that estimates the position of the vehicle or acquired states inside and outside the vehicle; A model selection unit that selects a second trained model based on the state, a model switching verification unit that compares a calculation result when the calculation is performed in parallel with the first trained model using the second trained model with a calculation result of the first trained model, and determines whether to switch from the first trained model to the second trained model based on the comparison result; The calculation unit switches from the first trained model to the second trained model when the model switching verification unit determines to switch to the second trained model, and performs the calculation; Further, a difference extraction unit that extracts a channel of a difference between the first trained model and the second trained model from the second trained model, The calculation unit performs the calculation using a channel of the difference, an on-board arithmetic processing device, characterized in that, when the model switching verification unit determines to switch to the second trained model, the calculation unit performs the calculation using a model of the first trained model excluding a channel that is a difference between the second trained model and the first trained model.

2. 2. The on-board processor according to claim 1, The vehicle-mounted arithmetic processing device, characterized in that the model switching verification unit includes a model accuracy calculation unit that calculates accuracy of the first trained model and the second trained model, with values ​​obtained based on different sensor information as correct recognition results.

3. 2. The on-board processor according to claim 1, The model switching verification unit includes a model accuracy calculation unit that calculates accuracy of a recognition result of a second trained model using a result of computing the first trained model as a correct recognition result, An in-vehicle computing device that determines, if the accuracy of a second trained model is equal to or greater than a threshold, to switch to a model with higher accuracy.

4. 4. The on-board processor according to claim 3, The model accuracy calculation unit includes a result prediction unit that predicts a current recognition result from a result of calculating the first trained model in the past, and calculating accuracy of a second model by using the recognition result predicted by the result prediction unit as a correct recognition result.

5. 2. The on-board processor according to claim 1, The model switching verification unit has a driving history reference unit that references a driving history such as past route information and driving time, 1. An in-vehicle calculation processing device, comprising: a controller for controlling a vehicle speed in accordance with a first embodiment of the present invention; a controller for controlling a vehicle speed in accordance with a second embodiment of the present invention;

6. 2. The on-board processor according to claim 1, The in-vehicle processing device is characterized in that the model selection unit selects a plurality of second trained models, and the model switching verification unit determines the most accurate second trained model from the plurality of second trained models based on a comparison result.

7. A calculation processing method that is installed in a vehicle and performs calculations using a trained model, A calculation step of performing a calculation using the first trained model; A state estimation step of estimating a position of the vehicle or an acquired state inside and outside the vehicle; A model selection step of selecting a second trained model based on the state; a model switching verification step of comparing a calculation result obtained by performing the calculation in parallel with the first trained model using the second trained model with a calculation result of the first trained model, and determining whether or not to switch from the first trained model to the second trained model based on the comparison result; a second calculation step of switching from the first trained model to the second trained model and performing the calculation when it is determined to switch to the second trained model in the model switching verification step, Further, a difference extraction step of extracting a channel of a difference between the first trained model and the second trained model from the second trained model, In the calculation step, the calculation is performed using the difference channel, A computation method characterized in that, in the model switching verification step, when it is decided to switch to the second trained model, the computation is performed using a model of the first trained model excluding a channel that is a difference between the second trained model and the first trained model.

Images