Vehicle data collection system and vehicle data collection method

Abstract

This vehicle data collection system comprises: a first data collection device for collecting, from a first vehicle, data with a feature amount output from a neural network intermediate layer corresponding to an AI failure; a second data collection device for detecting the occurrence of an event corresponding to the AI failure by using a classifier acquired from a server, and transmitting, to the server, vehicle data upon the occurrence of the event corresponding to the AI failure; and the server for creating the classifier capable of classifying a feature amount similar to the feature amount corresponding to the AI failure managed by a database update unit.

Description

Vehicle data collection system and vehicle data collection method

[0001] The present invention relates to a vehicle data collection system.

[0002] In recent years, the development of software-defined vehicles (SDVs) has attracted attention in the automotive field, leading to the continuous evolution of automotive software functions. In particular, vehicle data collection technologies are being investigated for the development of artificial intelligence (AI) that utilizes data obtainable from actual vehicles.

[0003] The following prior art exists as background art in this technical field: For example, Patent Document 1 (WO 2020 / 056331) describes a method including receiving sensor data, applying a neural network to the sensor data, applying a trigger classifier to an intermediate result of the neural network to determine a classifier score for the sensor data, and determining whether to transmit at least a portion of the sensor data over a computer network based at least in part on the classifier score.

[0004] Further, Patent Document 2 (JP 2023-163683 A) describes a vehicle control determination unit that creates a control signal to control the vehicle using data acquired from a sensor attached to the vehicle, and a near-miss detection unit that detects near-misses felt by the occupant from at least one of the biological changes of the occupant in the vehicle while driving the vehicle or the control signal of the vehicle, and outputs a near-miss detection signal. A near-miss detection unit, a near-miss detection signal, and a near-miss detection signal. A mismatch detection unit detects a mismatch between the control signal and the near-miss at the timing when the near-miss is detected from the data acquired from the sensor attached to the vehicle, the control signal created by the vehicle control determination unit, and the near-miss detection signal detected by the near-miss detection unit. A data storage unit that stores the data acquired from the sensor attached to the vehicle corresponding to the timing when the mismatch is detected by the determination mismatch detection unit and the control signal created by the vehicle control determination unit is described.

[0005] International Publication No. 2020 / 056331 Japanese Patent Application Laid-Open No. 2023-163683

[0006] According to Patent Document 1, when an AI-equipped vehicle faces a predefined scenario, vehicle data is collected as training data for the AI. According to Patent Document 2, vehicle data is saved on the vehicle when a mismatch occurs between the detection results of near misses by passengers and the vehicle control decision unit. These technologies enable developers to collect vehicle data related to scenarios that are difficult to anticipate in vehicle control software development, especially in AI development.

[0007] However, the function to detect these scenarios that are difficult to anticipate in advance increases hardware costs and software processing costs, making it difficult to install the function in inexpensive vehicles and limiting the number of vehicles from which data can be collected. It is necessary to collect vehicle data regarding scenarios that are difficult for AI developers to anticipate while suppressing increases in hardware costs and software processing costs.

[0008] The present invention aims to collect vehicle data regarding scenarios that are difficult to envision while suppressing increases in hardware costs and software processing costs.

[0009] A representative example of the invention disclosed in the present application is as follows: That is, a vehicle data collection system for collecting vehicle data includes a first data collection device for collecting data from a first vehicle, a second data collection device for collecting data from a second vehicle, and a server connected to the first data collection device and the second data collection device, wherein the first data collection device has a first AI execution unit for executing an in-vehicle AI, an AI fault detection unit for detecting a fault in the first AI execution unit, and an intermediate layer feature extraction unit for transmitting a feature output from a neural network intermediate layer corresponding to an AI fault detected by the AI ​​fault detection unit to a server, and the server The second data collection device has a database update unit that manages feature data collected from the first data collection device, and a classifier update unit that creates a classifier that can classify feature quantities similar to feature quantities corresponding to AI malfunctions managed by the database update unit and transmits the classifier to data collection device B, and the second data collection device has a second AI execution unit that executes in-vehicle AI, an AI malfunction equivalent detection unit that detects the occurrence of an event corresponding to an AI malfunction using the classifier acquired from the server, and an AI malfunction equivalent occurrence vehicle data collection unit that transmits vehicle data when the event corresponding to the AI ​​malfunction occurs to the server.

[0010] According to one aspect of the present invention, vehicle data can be collected at the time of occurrence of an AI malfunction. Objects, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0011] FIG. 1 is a diagram showing an overall configuration of a vehicle data collection system according to a first embodiment of the present invention. FIG. 2 is a diagram showing an example of a connection relationship between a data collection device B according to the first embodiment of the present invention and a control device connected via an in-vehicle network. FIG. 3 is a flowchart of processing executed by an AI execution unit of a data collection device A according to the first embodiment of the present invention. FIG. 4 is a diagram showing an example of an AI execution result in the data collection device A according to the first embodiment of the present invention. FIG. 5 is a diagram showing an example of a driver operation result in the data collection device A according to the first embodiment of the present invention. FIG. 6 is a flowchart of processing executed by an AI malfunction determination unit in the data collection device A according to the first embodiment of the present invention. FIG. 7 is a diagram showing an example of an AI malfunction detection result in the data collection device A according to the first embodiment of the present invention. FIG. 8 is a flowchart of processing executed by an NN intermediate layer feature collection unit in the data collection device A according to the first embodiment of the present invention. FIG. 9 is a flowchart of processing executed by a database update unit in a server according to the first embodiment of the present invention. FIG. 10 is a flowchart of processing executed by a classifier update unit in a server according to the first embodiment of the present invention. FIG. 11 is a flowchart of processing executed by an AI execution unit in a data collection device B according to the first embodiment of the present invention. FIG. 12 is a flowchart of processing executed by an AI malfunction equivalent detection unit in the data collection device B according to the first embodiment of the present invention. FIG. 10 is a flowchart of processing executed by a vehicle data collection unit when an AI malfunction equivalent occurs in data collection device B according to the first embodiment of the present invention. FIG. 11 is a diagram showing a detailed configuration of a vehicle data collection system 1 according to a second embodiment of the present invention. FIG. 12 is a diagram showing an example of classifier information according to the second embodiment of the present invention. FIG. 13 is a diagram showing an example of environmental data according to the second embodiment of the present invention. FIG. 14 is a flowchart of processing executed by a processing load measurement unit according to the second embodiment of the present invention. FIG. 15 is a flowchart of processing executed by a classifier selection unit according to the second embodiment of the present invention. FIG. 16 is a diagram showing a detailed configuration of a vehicle data collection system according to a third embodiment of the present invention. FIG. 17 is a flowchart of processing executed by a collected data evaluation unit according to the third embodiment of the present invention. FIG. 18 is a flowchart of processing executed by an NN intermediate layer designation unit according to the third embodiment of the present invention. FIG. 19 is a diagram showing the overall configuration of a vehicle data collection system according to a fourth embodiment of the present invention. FIG. 19 is a flowchart of processing executed by a classifier update determination unit according to the fourth embodiment of the present invention.

[0012] The present embodiment relates to a data collection device and a vehicle data collection system.

[0013] First Embodiment A vehicle data collection system according to a first embodiment of the present invention will be described with reference to FIGS.

[0014] FIG. 1 is a diagram showing the overall configuration of a vehicle data collection system 1 according to a first embodiment of the present invention.

[0015] The vehicle data collection system 1 of Example 1 includes a data collection device A2, a data collection device B3, and a server 4. The data collection device A2 and the data collection device B3 are installed in different types of vehicles. The data collection device A2 and the data collection device B3 can communicate with the server 4 via a wired or wireless connection. The data collection device A2 includes sensor data 101, an AI execution unit 102, an AI execution result 103, a driver operation result 104, an AI malfunction detection unit 105, an AI malfunction detection result 106, and an NN intermediate layer feature collection unit 107. The data collection device A2 has a so-called shadow mode function and verifies the AI ​​execution result 103 and the driver operation result 104 output from the AI ​​execution unit 102 for driving control in the background. The data collection device B3 includes sensor data 110, an AI execution unit 111, an AI malfunction equivalent detection unit 112, an AI malfunction equivalent detection result 113, and an AI malfunction equivalent occurrence vehicle data collection unit 114. The data collection device B3 differs from the data collection device A2 in that it does not have a shadow mode function, or has the shadow mode function but the shadow mode function is not operating. The server 4 has a database update unit 108, a classifier update unit 109, and AI learning data 115.

[0016] The data collection device A2 is an electronic control unit having an arithmetic device, a storage device, and a communication interface. The arithmetic device is a processor (e.g., a microcomputer) that executes programs stored in the storage device. The arithmetic device operates as a functional block that provides various functions by executing predetermined programs. The storage device includes a non-volatile storage area and a volatile storage area. The non-volatile storage area includes a program area accessible to the arithmetic device that stores programs executed by the arithmetic device, and a data area that temporarily stores data used by the arithmetic device when executing programs. The volatile storage area stores data used by the arithmetic device when executing programs. The communication interface connects to other electronic control units via a network such as CAN or Ethernet. The data collection device B3 may be configured with the same hardware as the data collection device A2.

[0017] The server 4 is configured by a computer having a processor (CPU), a memory, an auxiliary storage device, and a communication interface.

[0018] The processor of the server 4 is an arithmetic device that executes programs stored in memory. The processor executes various programs to realize the functions of each functional unit of the server 4. Note that some of the processing performed by the processor executing the programs may be executed by another arithmetic device (for example, hardware such as an ASIC or FPGA).

[0019] The memory of the server 4 includes ROM, which is a non-volatile storage element, and RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS), etc. The RAM is a high-speed, volatile storage element such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor and data used when the programs are executed.

[0020] The auxiliary storage device of the server 4 is, for example, a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or flash memory (SSD). The auxiliary storage device also stores data used by the processor when executing a program, and the program executed by the processor. In other words, the program is read from the auxiliary storage device, loaded into memory, and executed by the processor to realize each function of the server 4.

[0021] The communication interface of the server 4 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0022] The programs executed by the processor of the server 4 are provided to the server 4 via removable media (CD-ROM, flash memory, etc.) or a network, and are stored in a non-volatile auxiliary storage device, which is a non-transitory storage medium. For this reason, the server 4 should preferably have an interface for reading data from removable media.

[0023] The server 4 is a computer system configured on one physical computer or on multiple logically or physically configured computers, and may run on a virtual computer built on multiple physical computer resources. For example, multiple programs that realize the functions of the server 4 may each run on separate physical or logical computers, or multiple programs may be combined to run on a single physical or logical computer.

[0024] FIG. 2 is a diagram showing an example of the connection relationship between the data collection device B3 and the control devices connected via an in-vehicle network according to the first embodiment of the present invention.

[0025] The data collection device B3 is connected to the camera control device 5, radar control device 6, sonar control device 7, and gateway 8 via an in-vehicle network. The gateway 8 transmits vehicle data such as vehicle speed to the data collection device B3. The camera control device 5, radar control device 6, and sonar control device 7 transmit sensor data 110 collected by their respective sensors to the data collection device B3. The sensors may include LiDAR in addition to the sensors described above. The data collection device B3 determines whether to collect the vehicle data and sensor data 110 and transmits the target data determined to be collected to the server 4. As shown in the figure, the vehicle has multiple control devices and gateways 8 connected via an in-vehicle network, and the devices communicate sensor data 110, vehicle data, and the like to collect data. Communication over these networks occurs periodically or aperiodically. While FIG. 2 shows an example of the connection of the data collection device B3, the data collection device A2 may also be connected in a similar manner.

[0026] FIG. 3 is a flowchart of the process executed by the AI ​​execution unit 102 of the data collection device A2 according to the first embodiment of the present invention.

[0027] The AI ​​execution unit 102 executes AI based on sensor data 101 acquired from the camera control device 5, radar control device 6, sonar control device 7, etc., and outputs the execution result. That is, the AI ​​execution unit 102 inputs the sensor data 101 into a machine learning model configured by a neural network, and outputs the execution result output from the machine learning model to the outside. The AI ​​execution unit 102 also outputs feature quantities acquired from the intermediate layer of the neural network during the AI ​​execution process. For example, the AI ​​execution unit 102 acquires sensor data 101 in step S10202, and in step S10203, the model to which the sensor data 101 has been input detects an object and executes vehicle control processing. Then, in step S10204, the AI ​​execution unit 102 outputs the feature quantities acquired from the intermediate layer of the neural network. Then, in step S10205, the AI ​​execution result 103, which is a vehicle control command value, is output. The output AI execution result 103 may not be a vehicle control command value, but may be an AI recognition result or judgment result. In addition, any layer can be selected as the intermediate layer in the neural network.

[0028] FIG. 4 is a diagram showing an example of the AI ​​execution result 10301 in the data collection device A2 according to the first embodiment of the present invention.

[0029] The AI ​​execution result 10301 includes at least one of the AI ​​recognition result, judgment result, and control command value. For example, the illustrated AI execution result 10301 is the accelerator and brake control command value per second. The output period is variable. In addition to the accelerator and brake, the control command value information may also include information related to vehicle control such as steering.

[0030] FIG. 5 is a diagram showing an example of the driver operation result 10401 in the data collecting device A2 according to the first embodiment of the present invention.

[0031] The driver operation result 10401 includes at least one of the driver's operation results of the accelerator, brake, and steering, and is treated as a true value in the shadow mode. For example, the illustrated driver operation result 10401 is the operation result of the accelerator and brake. The acquisition period of the operation value may be fixed or variable.

[0032] FIG. 6 is a flowchart of the process executed by the AI ​​fault detection unit 105 in the data collection device A2 according to the first embodiment of the present invention.

[0033] The AI ​​malfunction detection unit 105 determines malfunctions that occur when AI is executed in a vehicle. Examples of AI malfunctions include erroneous recognition results, such as incorrectly identifying the class of a recognized object, and erroneous determinations that braking is required in a scene where braking is not required. For example, the AI ​​malfunction detection unit 105 acquires an AI execution result in step S10502 and acquires a driver operation result 104 in step S10503. Then, in step S10504, the AI ​​execution result 103 is compared with the driver operation result 104. If the AI ​​execution result 103 differs from the driver operation result 104, which is the true value, the AI ​​execution result 103 is incorrect. Therefore, the AI ​​malfunction occurrence is identified, and an AI malfunction detection result is output in step S10505. The vehicle control system stores the control command value, which is the AI ​​execution result 103, in a cache memory when transmitting it to an actuator, and compares the control command value stored in the cache memory with the driver operation result 104 obtained after AI execution. The control command value and the driver operation result 104 can be synchronized by time synchronization or sequence number synchronization. When comparing the control command value and the driver operation result 104, the driver operation result 104 is treated as a true value, and the difference between the driver operation result 104 and the AI ​​execution result 103 is determined using a predetermined threshold value, and if the difference is large, an AI malfunction is determined. In AI malfunction determination, the object to be compared with the AI ​​execution result may be, in addition to the driver operation result 104, the execution result of an AI with a different version or the execution result of an AI for another sensor. Furthermore, an AI malfunction may be determined in the server 4 rather than in the vehicle's data collection device A2. When AI malfunction determination is not performed, it is advisable to allocate the computational resources for the AI ​​malfunction determination process to other processes.

[0034] FIG. 7 is a diagram showing an example of an AI fault detection result 10601 in the data collection device A2 according to the first embodiment of the present invention.

[0035] The AI ​​malfunction detection result 10601 includes whether or not an AI malfunction has occurred and the time. For example, in FIG. 7 , at time 00:00:01:000, the accelerator control command value and brake control command value of the AI ​​execution result 103 match the accelerator operation result and brake operation result of the driver operation result, and a determination result of no AI malfunction is stored. On the other hand, in FIG. 7 , from time 00:00:02:000 to 00:00:04:000, the accelerator control command value and brake control command value of the AI ​​execution result 103 differ from the accelerator operation result and brake operation result of the driver operation result, and a determination result of the presence of a malfunction is stored. As described above, the presence or absence of an AI malfunction may be determined using a predetermined threshold. Furthermore, the cycle for storing the determination results may be fixed or variable.

[0036] FIG. 8 is a flowchart of the process executed by the NN hidden layer feature value collecting unit 107 in the data collecting device A2 according to the first embodiment of the present invention.

[0037] The NN intermediate layer feature collection unit 107 collects features in the neural network corresponding to the occurrence of an AI malfunction. For example, the NN intermediate layer feature collection unit 107 acquires the AI ​​malfunction detection result 106 in step S10702, extracts features output from the intermediate layer in the neural network corresponding to the detected AI malfunction in step S10703, and uploads the extracted features to the server 4 in step S10704. The intermediate layer in the neural network from which the features are extracted is variable and can be specified from a control device in the vehicle or a server 4 outside the vehicle. The features in the neural network during AI execution are temporarily stored when the AI ​​malfunction detection result 106 is acquired until it is determined whether feature extraction is necessary. If it is difficult to upload the extracted features to the server 4, they may be temporarily stored in the vehicle and uploaded to the server 4 when uploading becomes possible. The NN intermediate layer feature collection unit 107 is a computer that extracts predetermined features and may be provided outside the data collection device A2.

[0038] FIG. 9 is a flowchart of the process executed by the database update unit 108 in the server 4 according to the first embodiment of the present invention.

[0039] For example, in step S10802, the database update unit 108 receives features output from an intermediate layer in the neural network and transmitted from the vehicle, and in step S10803, stores the received features in storage in the server, thereby updating the database provided in the database update unit 108. The server 4 receives features from multiple vehicles. The database stores feature data related to the occurrence of AI malfunctions.

[0040] FIG. 10 is a flowchart of the process executed by the classifier update unit 109 in the server 4 according to the first embodiment of the present invention.

[0041] The classifier update unit 109 groups multiple feature quantities related to AI malfunctions in the database and creates a classifier that can determine which AI malfunction group a new feature quantity belongs to or does not belong to. The created classifier is distributed to each vehicle and updates the classifier of the AI ​​malfunction equivalent detection unit 112. For example, in step S10902, the classifier update unit 109 accesses a database on a server in which feature quantities are stored and acquires feature quantities output from the intermediate layer in the event of an AI malfunction. Then, in step S10903, the classifier is created using the feature quantities acquired from the database. For example, by clustering the feature quantities acquired from the database and determining the feature quantities of the cluster, a classifier that determines feature quantities within a predetermined range from the feature quantities of the cluster can be created. Then, in step S10904, the created classifier is distributed to the data collection device B3, and the classifier of the AI ​​malfunction equivalent detection unit 112 is updated. The classifier may be distributed to the data collection device A2 for update. Furthermore, when creating a classifier, not only feature quantities in the event of an AI malfunction but also feature quantities in the event of other events may be used.

[0042] FIG. 11 is a flowchart of the process executed by the AI ​​execution unit 111 in the data collection device B3 according to the first embodiment of the present invention.

[0043] The AI ​​execution unit 111 executes AI based on sensor data acquired from the camera control device 5, radar control device 6, sonar control device 7, etc., and outputs feature amounts acquired from the intermediate layer of the neural network during the AI ​​execution process. For example, the AI ​​execution unit 111 acquires sensor data 110 in step S11102. Then, in step S11103, the model to which the sensor data 110 has been input detects an object and executes vehicle control processing. Then, in step S11104, it outputs feature amounts acquired from the intermediate layer of the neural network during the AI ​​execution process. It may also output recognition results, judgment results, and vehicle control command values, which are the results of AI execution.

[0044] FIG. 12 is a flowchart of the process executed by the AI ​​malfunction equivalent detection unit 112 in the data collection device B3 according to the first embodiment of the present invention.

[0045] The AI ​​malfunction equivalent detection unit 112 uses a classifier acquired from the server 4 to classify the feature values ​​output from the intermediate layer of the neural network during AI execution. The classifier can classify the feature values ​​into those corresponding to an AI malfunction and those not corresponding to an AI malfunction based on the similarity of the feature values. For example, the AI ​​malfunction equivalent detection unit 112 acquires a classifier from the server 4 in step S11202, classifies the feature values ​​output from the intermediate layer in the neural network using the classifier in step S11203 to determine whether an AI malfunction is equivalent, and outputs the AI ​​malfunction equivalent determination result in step S11204. The classifier can be acquired from the server 4 at any timing. The neural network intermediate layer to be classified can be any layer and can be specified from a control device in the vehicle or the server 4 outside the vehicle.

[0046] FIG. 13 is a diagram showing an example of an AI malfunction equivalent detection result 10901 in the data collection device B3 according to the first embodiment of the present invention.

[0047] The AI ​​malfunction equivalent detection result 10901 includes whether or not an AI malfunction equivalent has occurred and the time. For example, in the example shown in FIG. 13 , from time 00:00:01:000 to 00:00:03:000, the feature values ​​output from the intermediate layer of the neural network during AI execution do not match the feature values ​​when an AI malfunction equivalent has occurred, so a result of no AI malfunction equivalent is stored. On the other hand, at time 00:00:04:000, the feature values ​​output from the intermediate layer of the neural network during AI execution match the feature values ​​when an AI malfunction equivalent has occurred, so a result of an AI malfunction equivalent is stored. The judgment and output period may be fixed or variable.

[0048] FIG. 14 is a flowchart of a process executed by the vehicle data collection unit 114 when an AI malfunction occurs in the data collection device B3 according to the first embodiment of the present invention.

[0049] The vehicle data collection unit 114 at the time of occurrence of an AI malfunction equivalent extracts vehicle data corresponding to the occurrence of an AI malfunction equivalent and transmits the extracted vehicle data to the server 4. For example, the vehicle data collection unit 114 at the time of occurrence of an AI malfunction equivalent acquires the AI ​​malfunction equivalent detection result 113 in step S11402, and accesses the sensor data 110 used for AI execution in step S11403. The sensor data 110 may be stored in a buffer for a predetermined time. Then, in step S11404, the sensor data 110 input to the AI ​​execution unit 111 at the time of the occurrence of an AI malfunction equivalent is extracted using the AI ​​malfunction equivalent detection time, and in step S11405, the extracted sensor data 110 and the AI ​​malfunction equivalent detection result 113 are transmitted to the server 4. The sensor data 110 accessed in step S11403 is temporarily stored on the vehicle until the data extraction determination and extraction are completed. The sensor data 110 to be extracted may include not only the sensor data 110 acquired from the camera control device 5, but also data handled by the vehicle, such as the sensor data 110 acquired from the radar control device 6 or the sensor data 110 acquired from the sonar control device 7. Furthermore, the vehicle data collection unit 114 at the time of occurrence of an AI malfunction may transmit the sensor data 110 to the server 4 at any timing.

[0050] As described above, in the vehicle data collection system 1 of this embodiment, the data collection device A2 detects an AI malfunction and transmits the corresponding feature data to the server 4. The server 4 then creates a classifier capable of classifying features similar to the collected features. The data collection device B3 then applies the classifier to detect an AI malfunction. This allows the data collection device B3 to collect vehicle data when an AI malfunction occurs without the AI ​​malfunction detection unit 105 of the data collection device A2, thereby reducing hardware and software processing costs. Comparing software execution results with driver responses, as in Patent Document 2, requires CPU (Central Processing Unit) resources for temporary data storage before comparison, as well as GPU (Graphics Processing Unit) resources for driver response detection. In the vehicle data collection system 1 of this embodiment, a classifier is executed on features in a neural network during AI execution, thereby minimizing the increase in processing load and enabling execution within existing GPU resources. This minimizes increases in hardware and software processing costs, making it possible to detect an AI malfunction and collect vehicle data.

[0051] <Embodiment 2> Selection and Execution of Classifier A vehicle data collection system according to embodiment 2 of the present invention will be described with reference to Figures 15 to 19. Embodiment 2 differs from embodiment 1 in that the classifier used is dynamically changed depending on the conditions inside and outside the vehicle when the AI ​​malfunction equivalent detection unit in the data collection device B3 is running. Embodiment 2 has the effect of reducing the number of classifiers applied when the AI ​​is running, thereby reducing the processing load due to classifier execution. Note that embodiment 2 will mainly describe the differences from embodiment 1, and the same configurations and processing procedures as embodiment 1 will be assigned the same reference numerals and their description will be omitted.

[0052] FIG. 15 is a diagram showing a detailed configuration of the vehicle data collection system 1 according to the second embodiment of the present invention.

[0053] The vehicle data collection system 1 of Example 2 includes a data collection device A2, a data collection device B3, and a server 4. The data collection device A2 includes sensor data 101, an AI execution unit 102, an AI execution result 103, a driver operation result 104, an AI malfunction detection unit 105, an AI malfunction detection result 106, and an NN intermediate layer feature collection unit 107. The data collection device B3 includes sensor data 110, an AI execution unit 111, an AI malfunction equivalent detection unit 112, an AI malfunction equivalent detection result 113, an AI malfunction equivalent occurrence vehicle data collection unit 114, environmental data 117, a processing load measurement unit 118, and a classifier selection unit 119. The server 4 includes a database update unit 108, a classifier update unit 109, AI learning data 115, and classifier information 116.

[0054] FIG. 16 is a diagram illustrating an example of the classifier information 11601 according to the second embodiment of the present invention.

[0055] The classifier information 11601 includes conditions under which the classifier is suitable for execution and a priority for applying the classifier. For example, it includes the vehicle, location, weather, time, etc. that the classifier can handle. Location information includes information such as vehicle coordinate information and road type. Time period information includes date information. The server 4 creates multiple types of classifiers and has metadata for each classifier. The priority is referenced when selecting the classifier to run, and is set on the server 4 side in the order of the environment in which data is to be collected.

[0056] The classifier update unit 109 acquires from the data collection device A2 the features output from the intermediate layer in the neural network, as well as environmental information related to the features when the AI ​​malfunctions. The acquired environmental information is then clustered to determine the environmental information to be determined by the classifier. Note that some classifiers can handle a variety of environments, and the environment may be indefinite.

[0057] FIG. 17 is a diagram showing an example of the environment data 11701 according to the second embodiment of the present invention.

[0058] The environmental data 11701 includes the type of data and the content represented by the data relating to the vehicle equipped with the data collection device B and the vehicle driving environment. The environmental data 11701 includes, for example, the vehicle type represented based on the VIN (Vehicle Identification Number), location information represented based on vehicle coordinate information obtained by GPS and the type of road the vehicle is traveling on recognized by various sensors, the recognition results of various sensors, weather information represented based on server-acquired information, and time zone information including date and time information held by the vehicle.

[0059] FIG. 18 is a flowchart of the process executed by the processing load measuring unit 118 according to the second embodiment of the present invention.

[0060] For example, the processing load measurement unit 118 accesses each control unit in the data collection device in step S11802, measures the processing load of each control unit in step S11803, and outputs the measured processing load in step S11804. The processing load measured by the processing load measurement unit 118 is a value related to the processing performance of the data collection device B3, such as the CPU, GPU, memory, and network bandwidth.

[0061] FIG. 19 is a flowchart of the process executed by the classifier selection unit 119 according to the second embodiment of the present invention.

[0062] The classifier selection unit 119 selects, from among multiple classifiers obtainable from the server 4, a classifier that is likely to detect features corresponding to an occurrence of an AI execution malfunction, depending on the vehicle type and driving environment, and activates the selected classifier. The classifier selection unit 119 also adjusts the number of classifiers to be activated depending on the processing load of the data collection device so as not to overload processing resources. For example, the classifier selection unit 119 obtains classifier information 11601 in step S11902, obtains environmental data 11701 in step S11903, and obtains processing load information of the data collection device in step S11904. The number of classifiers that can operate simultaneously is limited by the processor installed in the data collection device. Then, in step S11905, the classifier that is suitable for the environmental data 11701 from the classifier information 11601 is set to be active. For example, if all classifiers are operable based on the processing load information of the data collection device, all classifiers are activated. On the other hand, when the number of classifiers to be operated is limited, classifiers that are suitable for the environment are operated in descending order of priority.

[0063] As described above, in the vehicle data collection system 1 of the second embodiment of the present invention, the data collection device B3 has the environmental data 117, the processing load measurement unit 118, and the classifier selection unit 119, and the server 4 has the classifier information 116. Therefore, while maintaining the number of scenarios equivalent to defects that can be detected by the classifier in the data collection device B3, the number of classifiers to be activated can be reduced, thereby achieving the effect of reducing the processing load.

[0064] <Example 3> Instruction of feature extraction target A vehicle data collection system according to Example 3 of the present invention will be described with reference to Figures 20 to 22. Example 3 differs from Example 1 in that the server 4 evaluates whether the collected AI learning data 115 corresponds to an AI malfunction, and specifies an intermediate layer when the data collection device A2 extracts features based on the evaluation result. Example 3 will mainly describe the differences from Example 1, and the same configurations and processing procedures as Example 1 will be assigned the same reference numerals and their description will be omitted.

[0065] FIG. 20 is a diagram showing a detailed configuration of a vehicle data collection system 1 according to a third embodiment of the present invention.

[0066] The vehicle data collection system 1 of Example 3 includes a data collection device A2, a data collection device B3, and a server 4. The data collection device A2 includes sensor data 101, an AI execution unit 102, an AI execution result 103, a driver operation result 104, an AI malfunction detection unit 105, an AI malfunction detection result 106, and an NN intermediate layer feature collection unit 107. The data collection device B3 includes sensor data 110, an AI execution unit 111, an AI malfunction equivalent detection unit 112, an AI malfunction equivalent detection result 113, and an AI malfunction equivalent occurrence vehicle data collection unit 114. The server 4 includes a database update unit 108, a classifier update unit 109, AI learning data 115, a collected data evaluation unit 120, and an NN intermediate layer designation unit 121.

[0067] FIG. 21 is a flowchart of the process executed by the collected data evaluation unit 120 according to the third embodiment of the present invention.

[0068] The collected data evaluation unit 120 evaluates whether the collected AI learning data 115 corresponds to an AI malfunction. For example, the collected data evaluation unit 120 accesses the AI ​​learning data 115 in step S12002, references the evaluation target data in step S12003, evaluates whether the evaluation target data is involved in the AI ​​malfunction in step S12004, and outputs the evaluation result in step S12005. The AI ​​learning data 115 is, for example, camera image data, and includes information at the time of AI execution and information at the time of classifier execution as metadata, such as the collection time, the version of the AI ​​executed, the type of AI malfunction that was detected, and the type of classifier executed. Furthermore, the information at the time of classifier execution includes information on which layer in the neural network the classifier was applied to the feature extracted from. In step S12003, the metadata of the AI ​​learning data 115 is referenced to select a specific evaluation target. In step S12004, based on the AI ​​learning data 115 and its metadata, an automatic or manual evaluation is performed to determine whether the data is involved in the AI ​​malfunction. When an automatic evaluation is performed, for example, the degree of involvement is evaluated using a machine learning model for evaluation trained with the AI ​​learning data 115. When a manual evaluation is performed, the evaluator visually compares the image, the AI ​​execution results, and the hit information equivalent to the AI ​​malfunction with the true value to evaluate the degree of involvement.

[0069] FIG. 22 is a flowchart of the process executed by the NN hidden layer designation unit 121 according to the third embodiment of the present invention.

[0070] The NN intermediate layer designation unit 121 designates a neural network intermediate layer for extracting features in the data collection device A2 based on the evaluation results of the AI ​​learning data 115 acquired from the collected data evaluation unit 120. For example, the NN intermediate layer designation unit 121 acquires the collected data evaluation results in step S12102, determines a neural network intermediate layer for extracting features in step S12103, and delivers the determined intermediate layer information to the data collection device A2 in step S12104. In step S12103, if the acquired evaluation results have a low degree of involvement in the AI ​​malfunction, the layer for extracting features may be designated as a deep layer closer to the output side to increase the degree of involvement. Alternatively, if the acquired evaluation results have a high degree of involvement in the AI ​​malfunction, the layer for extracting features may be designated as a shallow layer closer to the input side to increase the degree of involvement.

[0071] As described above, in the vehicle data collection system 1 of this embodiment, the server 4 includes the collected data evaluation unit 120 and the NN intermediate layer designation unit 121, so that the neural network intermediate layer used for feature extraction can be designated depending on the degree of AI defect involvement in the collected data. When the degree of AI defect involvement is low, a deeper layer of the neural network intermediate layer is designated to collect features, thereby improving the accuracy of the created classifier in detecting AI defects. Furthermore, when the degree of AI defect involvement is high, a shallower layer of the neural network intermediate layer is designated to collect features, thereby reducing the processing load when the created classifier is executed. Therefore, the effect of increasing the degree of AI defect involvement in the collected data while reducing the processing load on the data collection device A2 is achieved.

[0072] <Fourth Embodiment> Classifier Update Timing Determination A vehicle data collection system according to a fourth embodiment of the present invention will be described with reference to Fig. 23 and Fig. 24. The fourth embodiment differs from the first embodiment in that the classification reception timing is determined according to the internal state of the data collection device B3. In the fourth embodiment, differences from the first and second embodiments will be mainly described, and the same configurations and processing procedures as those in the first and second embodiments will be assigned the same reference numerals and descriptions thereof will be omitted.

[0073] FIG. 23 is a diagram showing the overall configuration of a vehicle data collection system 1 according to a fourth embodiment of the present invention.

[0074] The vehicle data collection system 1 of Example 4 includes a data collection device A2, a data collection device B3, and a server 4. The data collection device A2 includes sensor data 101, an AI execution unit 102, an AI execution result 103, a driver operation result 104, an AI malfunction detection unit 105, an AI malfunction detection result 106, and an NN intermediate layer feature collection unit 107. The data collection device B3 includes sensor data 110, an AI execution unit 111, an AI malfunction equivalent detection unit 112, an AI malfunction equivalent detection result 113, an AI malfunction equivalent occurrence vehicle data collection unit 114, environmental data 117, a processing load measurement unit 118, and a classifier update determination unit 122. The server 4 includes a database update unit 108, a classifier update unit 109, and AI learning data 115. The environmental data 117 and the processing load measurement unit 118 perform the same processing as in Example 2 described above.

[0075] FIG. 24 is a flowchart of the process executed by the classifier update determination unit 122 in the fourth embodiment of the present invention.

[0076] The classifier update determination unit 122 updates the classifier in accordance with the vehicle state at a timing that does not affect the on-board software. For example, the classifier update determination unit 122 queries the server 4 for the classifier information 116 in step S12202 and checks whether a new version of the classifier is available on the server in step S12203. If a new version of the classifier is available, the classifier update determination unit 122 acquires the environmental data 117 in step S12204, acquires data collection device processing load information in step S12205, and determines whether the classifier can be updated using the information acquired from the server 4 in step S12206. For example, if the vehicle is located at home and Wi-Fi communication is available, it may be determined that the classifier can be updated. Alternatively, if the vehicle is stopped and the processing load is low, it may be determined that the classifier can be updated. If the classifier can be updated, the classifier update determination unit 122 requests the server 4 to update the classifier in step S12207. The query to the server 4 for the classifier information 116 in step S12202 may be performed periodically or event-driven in response to a predetermined event. Event-driven execution may be performed, for example, when a new classifier update is available, as obtained from the classifier update unit 109, or when the frequency of data collection by the vehicle data collection unit 114 is low when an AI malfunction occurs. The determination of whether the classifier can be updated in step S12206 may be made by combining the environmental data 117 and data collection device processing load information and taking into account the vehicle state. For example, the classifier may be updated when the vehicle system is started or shut down while the vehicle is parked in a parking lot. Furthermore, the vehicle state may take into account the load information of the data collection device or the load information of other vehicle control devices. The classifier may be updated when the additional information is lower than a predetermined threshold.

[0077] As described above, in the vehicle data collection system 1 of this embodiment, the data collection device B3 has the environmental data 117, the processing load measurement unit 118, and the classifier update determination unit 122, so that the classifier can be updated without affecting the on-board software. Furthermore, a classifier update event is executed based on an event that triggers a classifier update from the server 4 and the vehicle, and the latest classifier is used, thereby achieving the effect of efficient data collection.

[0078] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0079] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0080] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, hard disk, or SSD (Solid State Drive), or in a recording medium such as an IC card, SD card, or DVD.

[0081] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines that are necessary for implementation. In reality, it can be considered that almost all components are interconnected.

Claims

1. A vehicle data collection system that collects vehicle data, A first data acquisition device that collects data from the first vehicle, A second data acquisition device that collects data from the second vehicle, The system comprises the first data acquisition device and a server connected to the second data acquisition device, The first data acquisition device is The first AI execution unit that runs the in-vehicle AI, An AI malfunction detection unit for detecting a malfunction in the first AI execution unit, The system includes an intermediate layer feature extraction unit that transmits feature quantities output from the intermediate layer of the neural network corresponding to the AI ​​defect detected by the AI ​​defect detection unit to a server. The aforementioned server, A database update unit that manages feature data collected from the first data acquisition device, The system includes a classifier update unit that creates a classifier capable of classifying feature quantities similar to those corresponding to AI defects managed by the database update unit, and transmits it to a second data collection device. The second data acquisition device is The second AI execution unit executes the in-vehicle AI, An AI malfunction equivalent detection unit that detects the occurrence of an event equivalent to an AI malfunction using a classifier acquired from the server, A vehicle data collection system characterized by having an AI malfunction equivalent occurrence vehicle data collection unit that transmits vehicle data to a server when an event equivalent to the AI ​​malfunction occurs.

2. A vehicle data collection system according to claim 1, The second data acquisition device is A processing load measurement unit for measuring the processing load of the second data acquisition device, A vehicle data acquisition system characterized by having a classifier selection unit that selects a classifier that is effective in setting within the allowable range of the processing load, and for which the environmental conditions inside and outside the vehicle are suitable.

3. A vehicle data collection system according to claim 1, The aforementioned server, The collected data evaluation unit evaluates whether the collected data was related to the AI ​​malfunction, A vehicle data collection system characterized by having an NN intermediate layer designation unit that, based on the evaluation results of the collected data evaluation unit, designates the neural network intermediate layer to be featured in such a way that the involvement of the intermediate layer in AI malfunctions is increased.

4. A vehicle data collection system according to claim 1, The second data acquisition device is A vehicle data collection system characterized by having a classifier update determination unit that determines whether to update the classifier using the communication environment and processing load.

5. A vehicle data collection method performed by a vehicle data collection system, The vehicle data collection system comprises a first data collection device that collects data from a first vehicle, a second data collection device that collects data from a second vehicle, and a server connected to the first data collection device and the second data collection device. The aforementioned vehicle data collection method is: The first data acquisition device performs a first AI execution procedure for executing an in-vehicle AI, The first data acquisition device includes an AI malfunction detection procedure for detecting a malfunction in the first AI execution procedure, The first data acquisition device performs an intermediate layer feature extraction procedure, in which it transmits the feature quantities output from the intermediate layer of the neural network corresponding to the AI ​​defect detected in the AI ​​defect detection procedure to the server. The server performs a database update procedure for managing the feature data collected from the first data collection device, The server creates a classifier capable of classifying features similar to those corresponding to AI defects managed in the database update procedure, and transmits it to the second data collection device in a classifier update procedure. The second data acquisition device performs a second AI execution procedure for executing the in-vehicle AI, The second data acquisition device performs an AI malfunction equivalent detection procedure, which uses a classifier acquired from the server to detect the occurrence of an event equivalent to an AI malfunction, A vehicle data collection method characterized by comprising a procedure for collecting vehicle data when an event equivalent to the AI ​​malfunction occurs, wherein the second data collection device transmits vehicle data to a server when an event equivalent to the AI ​​malfunction occurs.