Data collection method, center, data collection program, and data collection system

Abstract

This center (3) identifies a scene in which a machine learning model does not satisfy a preset evaluation criterion on the basis of a result of model evaluation obtained by executing machine learning on the machine learning model using stored vehicle data, generates collection condition data indicating a collection condition including the collection data item and the collection start condition for the identified scene, and stores the generated collection condition data in association with scene identification information for identifying the scene. The center distributes the collection condition data to a plurality of on-vehicle devices (2, 200, 300). The center stores the vehicle data uploaded from the plurality of on-vehicle devices in association with the scene identification information.

Description

Data collection method, center, data collection program, data collection system CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This international application claims the benefit of Japanese Patent Application No. 2023-223211, filed with the Japan Patent Office on December 28, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to techniques for collecting data from multiple vehicles.

[0003] Patent Document 1 describes a system that collects vehicle data from a plurality of on-board devices mounted on a plurality of vehicles.

[0004] JP 2023-84379 A

[0005] As a result of detailed investigations by the inventors, it was found that, since a vehicle handles a wide variety of data, it is necessary to efficiently collect data from a plurality of on-board devices.

[0006] The present disclosure efficiently collects data from multiple in-vehicle devices.

[0007] One aspect of the present disclosure is a data collection method executed at a center configured to receive vehicle data including at least information about the vehicles from a plurality of on-board devices installed in each of a plurality of vehicles.

[0008] The center stores the vehicle data received from the multiple vehicle-mounted devices.

[0009] The center uses the stored vehicle data to perform machine learning including annotation, model training, and model evaluation on the machine learning model, and based on the results of model evaluation obtained, identifies scenes in which the machine learning model does not meet pre-set evaluation criteria.For the identified scenes, the center generates collection condition data indicating collection conditions including collection data items indicating the type of vehicle data to be collected and collection start conditions for starting data collection, and stores the generated collection condition data in association with scene identification information that identifies the scene.

[0010] The center distributes the collection condition data to a plurality of vehicle-mounted devices.

[0011] The center stores the vehicle data uploaded from the multiple vehicle-mounted devices based on the collection condition data, in association with the scene identification information.

[0012] The data collection method of the present disclosure configured as above can collect vehicle data by narrowing down the collected data by scene, thereby efficiently collecting data from multiple in-vehicle devices. Furthermore, the data collection method of the present disclosure can generate collection condition data by identifying scenes that do not satisfy the evaluation criteria, thereby efficiently generating collection condition data.

[0013] Another aspect of the present disclosure is a center configured to receive vehicle data including at least information about the vehicles from a plurality of on-board devices installed in each of a plurality of vehicles, the center comprising a data storage unit, a collection condition generation unit, and a distribution unit.

[0014] The data storage unit is configured to store the vehicle data received from the plurality of in-vehicle devices.

[0015] The collection condition generation unit is configured to identify scenes in which the machine learning model does not satisfy preset evaluation criteria based on the results of model evaluation obtained by performing machine learning including annotation, model training, and model evaluation on the machine learning model using the stored vehicle data, generate collection condition data for the identified scenes indicating collection conditions including collection data items indicating the type of vehicle data to be collected and collection start conditions for starting data collection, and store the generated collection condition data in association with scene identification information that identifies the scene.

[0016] The distribution unit is configured to distribute the collection condition data to a plurality of in-vehicle devices.

[0017] The data storage unit is configured to store the vehicle data uploaded from the plurality of vehicle-mounted devices based on the collection condition data in association with the scene identification information.

[0018] The center of the present disclosure is a device that executes the data collection method of the present disclosure, and can obtain the same effects as the data collection method of the present disclosure.

[0019] Yet another aspect of the present disclosure is a data collection program that causes a center computer configured to receive vehicle data including at least information about the vehicles from a plurality of on-board devices mounted on each of a plurality of vehicles to function as a data storage unit, a collection condition generation unit, and a distribution unit, and the data storage unit is configured to store the vehicle data uploaded from the plurality of on-board devices based on the collection condition data in association with scene identification information.

[0020] A computer controlled by the data collection program of the present disclosure can constitute a part of the center of the present disclosure, and can obtain the same effects as the center of the present disclosure.

[0021] Yet another aspect of the present disclosure is a data collection system including a plurality of vehicle-mounted devices and a center.

[0022] The center includes a data storage unit, a collection condition generation unit, and a distribution unit. The data storage unit is configured to store the vehicle data uploaded from the multiple in-vehicle devices based on the collection condition data, in association with the scene identification information.

[0023] The vehicle-mounted devices each include a collection unit and an upload unit.

[0024] The data collection system of the present disclosure is a system that executes the data collection method of the present disclosure, and can obtain the same effects as the data collection method of the present disclosure.

[0025] Yet another aspect of the present disclosure is a data collection method executed in a data collection system including a plurality of vehicle-mounted devices and a center.

[0026] The center stores vehicle data received from multiple on-board devices. The center uses the stored vehicle data to perform machine learning, including annotation, model training, and model evaluation, on a machine learning model, and based on the results of model evaluation obtained, identifies scenes in which the machine learning model does not satisfy preset evaluation criteria. For the identified scenes, the center generates collection condition data indicating collection conditions, including collection data items indicating the type of vehicle data to be collected and collection start conditions for starting data collection. The center stores the generated collection condition data in association with scene identification information that identifies the scenes. The center distributes the collection condition data to the multiple on-board devices. The center stores vehicle data uploaded from the multiple on-board devices based on the collection condition data in association with the scene identification information.

[0027] When the in-vehicle devices receive the collection condition data from the center, they collect vehicle data corresponding to the collection data items based on the collection conditions indicated by the collection condition data when a collection start condition is met. The in-vehicle devices associate the collected vehicle data with the collection conditions and upload it to the center.

[0028] The data collection method of the present disclosure is a method executed by the data collection system of the present disclosure, and by executing this method, it is possible to obtain the same effects as the data collection system of the present disclosure.

[0029] 1 is a block diagram showing the configuration of a data collection system of a first embodiment; FIG. 2 is a block diagram showing the configuration of a data collection device; FIG. 3 is a block diagram showing the configuration of a center; FIG. 4 is a functional block diagram showing the functional configuration of the center; FIG. 5 is a diagram explaining management of collection condition data and collected data; FIG. 6 is a diagram explaining traceability of an AI model; FIG. 7 is a flowchart showing collected data management processing; FIG. 8 is a flowchart showing the first half of model learning processing; FIG. 9 is a flowchart showing the second half of model learning processing; FIG. 10 is a flowchart showing collection condition generation processing; FIG. 11 is a flowchart showing distribution processing; FIG. 12 is a flowchart showing data collection processing; FIG. 13 is a block diagram showing the configuration of a data collection system of a second embodiment;

[0030] First Embodiment A first embodiment of the present disclosure will be described below with reference to the drawings.

[0031] As shown in FIG. 1, the data collection system 1 of this embodiment includes a plurality of data collection devices 2 and a center 3 .

[0032] The data collection device 2 is mounted on a vehicle and has a function of performing data communication with the center 3 via a wide area wireless communication network NW.

[0033] The center 3 is a device that manages the data collection system 1. The center 3 has a function of performing data communication with a plurality of data collection devices 2 via the wide area wireless communication network NW.

[0034] 2, the data collection device 2 includes a control unit 11, a CAN communication unit 12, a storage unit 13, and a communication unit 14. CAN is an abbreviation for Controller Area Network.

[0035] The control unit 11 is an electronic control device mainly composed of a microcomputer including a CPU 21, a ROM 22, a RAM 23, etc. The various functions of the microcomputer are realized by the CPU 21 executing a program stored in a non-transitory tangible recording medium. In this example, the ROM 22 corresponds to the non-transitory tangible recording medium storing the program. Furthermore, the execution of this program results in the execution of a method corresponding to the program. Note that some or all of the functions executed by the CPU 21 may be configured as hardware using one or more ICs, etc. Furthermore, the number of microcomputers constituting the control unit 11 may be one or more.

[0036] The CAN communication unit 12 is connected to a plurality of ECUs via communication lines so as to be able to communicate data with them, and transmits and receives data according to the CAN communication protocol. Specifically, the plurality of ECUs connected to the CAN communication unit 12 include an engine ECU that controls the engine, a brake ECU that controls the brakes, a steering ECU that controls the steering, a suspension ECU that controls the suspension, and an ECU that controls the on / off of lights. In Fig. 2, only ECUs 16, 17, and 18 are shown as ECUs connected to the CAN communication unit 12. ECU stands for Electronic Control Unit.

[0037] The storage unit 13 is a storage device for storing various data.

[0038] The communication unit 14 performs data communication with the center 3 via the wide area wireless communication network NW.

[0039] As shown in FIG. 3, the center 3 includes a control unit 31, a communication unit 32, and a storage unit 33.

[0040] The control unit 31 is an electronic control device mainly composed of a microcomputer including a CPU 41, a ROM 42, a RAM 43, etc. The various functions of the microcomputer are realized by the CPU 41 executing a program stored in a non-transitory tangible recording medium. In this example, the ROM 42 corresponds to the non-transitory tangible recording medium storing the program. Furthermore, the execution of this program results in the execution of a method corresponding to the program. Note that some or all of the functions executed by the CPU 41 may be configured as hardware using one or more ICs, etc. Furthermore, the number of microcomputers constituting the control unit 31 may be one or more.

[0041] The communication unit 32 performs data communication with a plurality of data collection devices 2 via the wide area wireless communication network NW.

[0042] The storage unit 33 is a storage device for storing various data, and includes a collection condition database (hereinafter referred to as a collection condition DB) 33a and a collection data database (hereinafter referred to as a collection data DB) 33b.

[0043] As shown in FIG. 4 , the center 3 includes functional blocks realized by the CPU 41 executing a program stored in the ROM 42, such as a data collection unit 101, a collected data and product storage unit 102, a collected data management and analysis unit 103, a machine learning unit 104, a collection condition management unit 105, a CI / CD unit 106, a UI unit 107, authentication and authorization units 108, 109, 110, 111, and 112, a collection condition file transmission unit 113, and an AI model file transmission unit 114. CI stands for Continuous Integration. CD stands for Continuous Delivery. UI stands for User Interface. AI stands for Artificial Intelligence.

[0044] The data collection unit 101 collects data from a plurality of data collection devices 2 mounted on a plurality of vehicles, respectively.

[0045] The collected data and product storage unit 102 stores the data collected by the data collection unit 101 and the products generated by the collected data management and analysis unit 103, the machine learning unit 104, and the collection condition management unit 105.

[0046] The collected data management and analysis unit 103 includes a preprocessing unit 121 and a collection status confirmation unit 122 .

[0047] The preprocessing unit 121 performs preprocessing (for example, changing the resolution, changing to grayscale, etc.) on the collected data to make it easier to use in machine learning.

[0048] The collection status checking unit 122 checks the amount of collected data.

[0049] The machine learning unit 104 includes an annotation unit 123 , a model training unit 124 , and a model evaluation unit 125 .

[0050] The annotation unit 123 performs annotation to add information (e.g., labels) for machine learning to the data that has been preprocessed by the preprocessing unit 121.

[0051] The model training unit 124 trains a machine learning model using the data generated by the annotation unit 123. The machine learning model is a model that uses, for example, image data captured by an in-vehicle camera as input data, determines objects in the image, and outputs the determination result.

[0052] The model evaluation unit 125 evaluates the accuracy of the machine learning model based on the training results of the model training unit 124.

[0053] The collection condition management unit 105 manages collection conditions used to collect data used for training a machine learning model. The collection condition management unit 105 includes a collection condition generation unit 126.

[0054] The collection condition generation unit 126 generates collection condition data indicating collection conditions for collecting data to improve the accuracy of the machine learning model, based on the evaluation results from the model evaluation unit 125.

[0055] The CI / CD unit 106 includes a collection condition build and distribution unit 127, an AI logic build and distribution unit 128, and a distribution status management unit 129.

[0056] The collection condition build and distribution unit 127 converts the collection condition data generated by the collection condition generation unit 126 into a format recognizable by the data collection device 2 and distributes it to the data collection device 2 .

[0057] The AI ​​logic build and distribution unit 128 converts the machine learning model (i.e., AI logic) generated by the machine learning unit 104 into a format that can be executed by the data collection device 2 and distributes it to the data collection device 2.

[0058] The distribution status management unit 129 checks the distribution status of the collection condition data and AI logic.

[0059] The UI unit 107 is a device for exchanging information between the data engineer, AI engineer, software engineer, test engineer, and operator and the center 3.

[0060] Data engineers are engineers who work on data used in machine learning. AI engineers are engineers who work on generating machine learning models. Software engineers are engineers who work on applications that are generated by combining multiple machine learning models. Test engineers are engineers who evaluate machine learning models and applications. Operators manage the distribution status of collection condition data and AI logic.

[0061] The authentication and authorization unit 108 performs authentication and authorization for access by a data engineer to the center 3. For example, the authentication and authorization unit 108 performs authentication and authorization so that the data engineer can access only the collected data management and analysis unit 103.

[0062] The authentication and authorization unit 109 performs authentication and authorization for access by the AI ​​engineer to the center 3. For example, the authentication and authorization unit 109 performs authentication and authorization so that the AI ​​engineer can access only the machine learning unit 104.

[0063] The authentication and authorization unit 110 performs authentication and authorization for software engineers' access to the center 3. For example, the authentication and authorization unit 110 performs authentication and authorization so that the software engineers can access only the machine learning unit 104.

[0064] The authentication and authorization unit 111 performs authentication and authorization for access by a test engineer to the center 3. For example, the authentication and authorization unit 111 performs authentication and authorization so that the test engineer can access only the collection condition management unit 105.

[0065] The authentication and authorization unit 112 performs authentication and authorization for an operator's access to the center 3. For example, the authentication and authorization unit 112 performs authentication and authorization so that the operator can access only the CI / CD unit 106.

[0066] The collection condition file transmission unit 113 transmits the collection condition file generated by the collection condition build / distribution unit 127 to a plurality of data collection devices 2 .

[0067] The AI ​​model file transmission unit 114 transmits the AI ​​model file generated by the AI ​​logic build and distribution unit 128 to multiple data collection devices 2.

[0068] The authentication and authorization units 108 to 112 are configured to be able to set different access rights for each of a plurality of applications (e.g., ADAS applications, multimedia applications) that use the machine learning models handled by the center 3. ADAS is an abbreviation for Advanced Driver Assistance System.

[0069] The collection conditions are generated manually by an engineer or automatically by an AI model, as shown by arrows L1 and L2 in FIG.

[0070] 5 includes conditions C1, C2, C3, etc. Conditions C1 and C2 are conditions for executing data collection. Data collection is executed when both conditions C1 and C2 are met.

[0071] Condition C3 indicates the timing for uploading the collected data to the center 3.

[0072] A collection condition profile is added to the collection condition file. The collection condition profile includes a collection condition ID and a scene tag. ID is an abbreviation for identification. "CJD-001" shown in Figure 5 is the collection condition ID. "'Night rain'" shown in Figure 5 is the scene tag. A scene includes external brightness, vehicle behavior, driver operation, road, application execution status, etc. Therefore, a collection condition ID corresponds to a scene. A scene tag is a character string set by an engineer to identify a scene.

[0073] The collection condition file to which the collection condition profile has been added is stored in the collection condition DB 33a, as indicated by arrow L3.

[0074] The collection condition file is distributed to the data collection device 2 mounted on the vehicle, as indicated by arrow L4.

[0075] The data collection device 2 collects at least one of image data and sensor data based on a collection condition file obtained by distribution from the center 3. Data D1 shown in Fig. 5 is image data. Data D2 shown in Fig. 5 is sensor data indicating the detection results of a sensor mounted on the vehicle.

[0076] The data collection device 2 adds a collection condition ID included in a collection condition profile corresponding to the collection condition file to the collected data. The data collection device 2 uploads the data with the collection condition ID added to the center 3, as indicated by arrow L5.

[0077] The center 3 stores the data uploaded from the data collection device 2 in the collected data DB 33b. As indicated by arrow L6, the data stored in the collected data DB 33b is linked to a scene by the collection condition ID. As a result, the data stored in the collected data DB 33b is linked to a scene tag (e.g., "night rain" in FIG. 5) corresponding to the collection condition ID.

[0078] The data stored in the collected data DB 33b is used by engineers as indicated by arrow L7.

[0079] As shown in Fig. 6, the collected data metadata D11 stores data classified according to a plurality of collection conditions. The collected data metadata D11 shown in Fig. 6 includes a first collection condition storage area R1 that stores data with a collection condition ID of "T-1" and a second collection condition storage area R2 that stores data with a collection condition ID of "T-2."

[0080] The collected data metadata D11 classifies and stores data for each of a plurality of vehicles in a collection condition storage area that stores data for each collection condition.

[0081] The first collection condition storage area R1 includes a first vehicle storage area R11 that stores data acquired from a vehicle identified by "VIN-1" and a second vehicle storage area R12 that stores data acquired from a vehicle identified by "VIN-2."

[0082] The second collection condition storage area R2 includes a third vehicle storage area R13 that stores data acquired from a vehicle identified by "VIN-3."

[0083] The vehicle storage area for saving data for each vehicle stores the data acquisition date and time, and a data ID and data type for each of one or more pieces of data.

[0084] The first vehicle storage area R11 indicates, for the vehicle identified by "VIN-1," the data acquisition date and time, the data type of data with a data ID of "Data-1" being sensor data, and the data type of data with a data ID of "Data-2" being image data.

[0085] The second vehicle storage area R12 indicates, for the vehicle identified by "VIN-2," the data acquisition date and time, the data type of data with a data ID of "Data-3" being sensor data, and the data type of data with a data ID of "Data-4" being image data.

[0086] The third vehicle storage area R13 indicates, for the vehicle identified by "VIN-3," the data acquisition date and time, the data type of data with a data ID of "Data-5" being sensor data, and the data type of data with a data ID of "Data-6" being image data.

[0087] The annotation job data D12 stores data classified for each of a plurality of annotation jobs.

[0088] The annotation job data D12 shown in FIG. 6 includes a first annotation job storage area R21 that stores data with the annotation job ID "Anot-1" and a second annotation job storage area R22 that stores data with the annotation job ID "Anot-2."

[0089] The annotation job storage area for storing data for each annotation job stores a collection condition ID and a data ID in association with each other for each collection condition ID.

[0090] The first annotation job storage area R21 stores the collection condition ID "T-1" in association with the data ID "Data-2", and also stores the collection condition ID "T-2" in association with the data ID "Data-6".

[0091] The second annotation job storage area R22 stores the collection condition ID "T-1" and the data ID "Data-4" in association with each other.

[0092] As shown by arrows L11 and L12, an annotation job with an annotation job ID of "Anot-1" uses image data collected under collection conditions specified by a collection condition ID of "T-1" and identified by a data ID of "Data-2", and image data collected under collection conditions specified by a collection condition ID of "T-2" and identified by a data ID of "Data-6".

[0093] As shown by arrow L13, an annotation job with an annotation job ID of "Anot-2" uses image data collected under collection conditions specified by a collection condition ID of "T-1" and specified by a data ID of "Data-4."

[0094] The model training job data D13 stores a training job ID and one or more annotation job IDs in association with each other.

[0095] The model training job data D13 shown in FIG. 6 stores a model training job ID of "Train-1," an annotation job ID of "Anot-1," and an annotation job ID of "Anot-2" in association with each other.

[0096] As shown by arrows L14 and L15, a model training job with a model training job ID of "Train-1" uses image data generated by an annotation job identified by an annotation job ID of "Anot-1" and image data generated by an annotation job identified by an annotation job ID of "Anot-2".

[0097] Next, a description will be given of the procedure of the collected data management process executed by the center 3. The collected data management process is a process that is repeatedly executed while the control unit 31 is operating.

[0098] When the collected data management process is executed, the CPU 41 of the control unit 31 determines whether new collected data has been saved in the collected data DB 33b in S10, as shown in Fig. 7. The collected data includes, for example, vehicle-mounted camera videos, vehicle-mounted camera still images, application operation results, sensor recognition results, and CAN data.

[0099] If no new collected data is stored in the collected data DB 33b, the CPU 41 ends the collected data management process. On the other hand, if new collected data is stored in the collected data DB 33b, the CPU 41 creates collected data metadata for the newly stored collected data in S20 and stores the created collected data metadata in the collected data DB 33b. The collected data metadata includes, for example, a data ID, collection date and time, data type (e.g., CAN, application operation, video, still image, sensor), collection condition ID, collected vehicle ID, and raw data storage path.

[0100] In S30, the CPU 41 performs preprocessing of the collected data. The preprocessing of the collected data may include, for example, changing the resolution, changing to grayscale, etc. As the preprocessing of the collected data, processing that takes privacy into consideration (for example, mosaic processing) may be performed.

[0101] In S40, the CPU 41 stores the preprocessed collected data (hereinafter, referred to as preprocessed collected data) in the collected data DB 33b. The collected data DB 33b includes a first data storage area for storing the collected data and a second data storage area for storing the preprocessed collected data.

[0102] In S50, the CPU 41 updates the collected data metadata. Specifically, the CPU 41 adds the storage path of the preprocessed collected data stored in S40 to the collected data metadata.

[0103] In S60, the CPU 41 counts the number of collected data for each scene. Specifically, the CPU 41 identifies the number of collected data stored in the collected data DB 33b for each scene based on the collection condition ID linked to the collected data.

[0104] In S70, the CPU 41 updates the collected data metrics based on the counting results of S60. The collected data metrics include, for example, a collection condition ID, date and time, and the number of collected data items. Specifically, the CPU 41 updates the date and time and the number of collected data items in the collected data metrics for each scene.

[0105] In S80, the CPU 41 checks the time from the start of collection to the present time (hereinafter, collection period) and the number of collected data for each collection condition.

[0106] The processes of S90 to S140 described below are performed for each of the multiple scenes.

[0107] In S90, the CPU 41 determines whether the collection period is equal to or longer than the generation determination time preset for each collection condition. If the collection period is shorter than the generation determination time, the CPU 41 ends the collected data management process. On the other hand, if the collection period is equal to or longer than the generation determination time, the CPU 41 determines in S100 whether the number of collected data items is less than the generation determination number preset for each collection condition.

[0108] Here, if the number of collected data is equal to or greater than the generation determination number, the CPU 41 ends the collected data management process. On the other hand, if the number of collected data is less than the generation determination number, the CPU 41 determines in S110 whether or not the collection conditions need to be relaxed. For example, if there is other data that can be included in the collection conditions, it is determined that the collection conditions need to be relaxed. Here, if the collection conditions need to be relaxed, the CPU 41 starts the collection condition generation process described below in S120 and ends the collected data management process.

[0109] On the other hand, if relaxation of the collection conditions is not necessary, the CPU 41 determines in S130 whether or not a change in the distribution settings is necessary. For example, if the vehicle types to be distributed or the areas to be distributed can be changed, it is determined that a change in the distribution settings is necessary. Here, if a change in the distribution settings is not necessary, the CPU 41 ends the collected data management process. On the other hand, if a change in the distribution settings is necessary, the CPU 41 starts a distribution setting generation process in S140 and ends the collected data management process. In the distribution setting generation process, the CPU 41 expands the areas to be distributed or increases the vehicle types to be distributed.

[0110] Next, we will explain the procedure of the model learning process executed by the center 3. The model learning process is started when the collected data used for learning the machine learning model is updated or when a preset execution period has elapsed.

[0111] When the model learning process is executed, the CPU 41 of the control unit 31 reads the annotation type included in the collection profile metadata of the collected data used to learn the machine learning model in S210, as shown in Fig. 8. The collection profile metadata includes a collection condition ID, creation date and time, collection start date and time, scene tag, annotation type, and collection condition status.

[0112] The scene tag can be "rainy" or "night", and can be set in combination. The annotation type indicates the type of annotation. The collection condition status indicates whether or not the information has been distributed to the vehicle.

[0113] At S220, the CPU 41 executes the annotation job based on the annotation type read at S210. By executing the annotation job, annotation results and annotation job data are generated. The annotation job data includes an annotation job ID, a start date and time, a pair of a data ID and a collection condition ID, a status (e.g., job in progress, success, failure), and a result storage path.

[0114] In S230, the CPU 41 checks the status included in the annotation job data (hereinafter, referred to as the annotation status).

[0115] In S240, the CPU 41 determines whether the annotation job is complete based on the confirmation result in S230. If the annotation job is not complete, the CPU 41 proceeds to S230. On the other hand, if the annotation job is complete, the CPU 41 updates the annotation status included in the annotation job data in S250.

[0116] In S260, the CPU 41 stores the annotation result in the storage unit 33.

[0117] In S270 , the CPU 41 stores the annotation job data in the storage unit 33 .

[0118] In S280, the CPU 41 determines whether the annotation result has been updated. If the annotation result has been updated, the CPU 41 proceeds to S300. On the other hand, if the annotation result has not been updated, the CPU 41 determines in S290 whether a preset training execution period has elapsed.

[0119] If the training execution period has not elapsed, the CPU 41 proceeds to S280. On the other hand, if the training execution period has elapsed, the CPU 41 proceeds to S300.

[0120] When proceeding to S300, the CPU 41 reads the model training settings. The model training settings include a training setting ID, a pair of required data type and number of data, and a path to a server setting file. The pair of required data type and number of data is information indicating the number of new data items required to start model training. The path to the server setting file is information indicating the computing environment used for model training.

[0121] 9, the CPU 41 determines whether the start condition indicated by the loaded model training setting is satisfied in S310. The start condition indicated by the model training setting is set by a pair of a required data type and the number of data.

[0122] If the start condition is not met, the CPU 41 repeats the process of S310 to wait until the start condition is met. When the start condition is met, the CPU 41 starts model training in S320. By executing model training, model training results and model training job data are generated. The model training job data includes a training job ID, start date and time, a list of annotation job IDs, a status (e.g., job in progress, success, failure), and a training result storage path.

[0123] In S330, the CPU 41 determines whether model training is complete. If model training is not complete, the CPU 41 repeats the process of S330 to wait until model training is complete. When model training is complete, the CPU 41 updates the status of the model training job data in S340.

[0124] In S350, the CPU 41 stores the model training results in the storage unit 33.

[0125] In S360 , the CPU 41 stores the annotation job data in the storage unit 33 .

[0126] The CPU 41 evaluates the model training results in S370. The evaluation of the model training results includes confirming whether the machine learning model is operating properly by using, as evaluation data, scenes in which the machine learning model did not operate properly, and detecting weaknesses in the machine learning model by adding noise to the annotation results or by cutting out parts of the images to confirm whether the machine learning model is operating properly.

[0127] In S380, the CPU 41 determines whether or not the collection conditions need to be updated based on the evaluation result in S370. For example, the CPU 41 determines that the collection conditions need to be updated when the machine learning model identifies a scene that does not satisfy a preset evaluation criterion.

[0128] If the collection conditions need to be updated, the CPU 41 starts a collection condition generation process (described later) at S390 and ends the model learning process. On the other hand, if the collection conditions do not need to be updated, the CPU 41 sets the machine learning model obtained by model training as a vehicle distributable model at S400 and ends the model learning process.

[0129] Next, we will explain the procedure of the collection condition generation process executed by the center 3. The collection condition generation process starts with the process of S120 or S390.

[0130] When the collection condition generation process is executed, the CPU 41 of the control unit 31 generates new data collection conditions in S510, as shown in FIG. 10 . Specifically, if the CPU 41 identifies a scene in which the machine learning model does not satisfy a preset evaluation criterion, the CPU 41 generates collection conditions for collecting the identified scene. Furthermore, if the CPU 41 detects a weakness in the machine learning model, the CPU 41 generates collection conditions for collecting data to overcome the weakness in the machine learning model. Furthermore, if the collection conditions need to be relaxed, the CPU 41 generates data collection conditions by relaxing the restrictions by changing thresholds such as the collection start period and the number of collected data.

[0131] In S520, the CPU 41 stores the collection condition file indicating the collection conditions generated in S510 in the collection condition DB 33a. As described above, a collection condition profile is added to the collection condition file.

[0132] In S530, the CPU 41 stores the collection profile metadata of the collection conditions generated in S510 in the collection condition DB 33a.

[0133] At S540, the CPU 41 determines whether the collection conditions generated at S510 are appropriate for distribution to a vehicle. If the collection conditions are not appropriate for distribution to a vehicle, the CPU 41 ends the collection condition generation process. On the other hand, if the collection conditions are appropriate for distribution to a vehicle, the CPU 41 sets the collection conditions generated at S510 as collection conditions that can be distributed to a vehicle at S550, and ends the collection condition generation process.

[0134] Next, a description will be given of the procedure of the distribution process executed by the center 3. The distribution process is started when the operator performs an input operation via the UI unit 107 to start the distribution process.

[0135] When the distribution process is executed, the CPU 41 of the control unit 31 sets the machine learning model and collection conditions to be distributed from among the machine learning models and collection conditions that can be distributed to the vehicle, based on the input operations performed by the operator via the UI unit 107, in S610, as shown in Figure 11.

[0136] In S620, the CPU 41 sets vehicles to be the distribution target (i.e., targets) using data for each vehicle such as the target vehicle model, target area, and frequency of use of functions.

[0137] In S630, the CPU 41 builds the machine learning model and collection conditions set in S610 for distribution.

[0138] In S640, the CPU 41 starts the process of distributing the machine learning model and collection conditions built in S630 to the targets set in S620.

[0139] The processes of S650 to S700, which will be described later, are performed for each of the multiple targets.

[0140] In S650, the CPU 41 inquires of the vehicle whether the vehicle is in a state in which installation is possible. Examples of states in which installation is not possible include a state in which there is insufficient storage space because another high-priority application is running while the vehicle is moving, or a state in which Wi-Fi communication is not possible. Wi-Fi is a registered trademark.

[0141] In S660, the CPU 41 determines whether the vehicle to be distributed is in an installable state. If the vehicle is not in an installable state, the CPU 41 repeats the process of S660 and waits until the vehicle is in an installable state.

[0142] Then, when the vehicle is ready for installation, the CPU 41 distributes the machine learning model and collection conditions set in S610 to the vehicle to be distributed in S670.

[0143] In S680, the CPU 41 checks the distribution processing status.

[0144] In S690, the CPU 41 determines whether or not the installation is complete based on the confirmation result in S680. If the installation is not complete, the CPU 41 proceeds to S680. On the other hand, if the installation is complete, the CPU 41 updates the distribution processing status in S700 and ends the distribution processing.

[0145] Next, a description will be given of the procedure of the data collection process executed by the control unit 11 of the data collection device 2. The data collection process is a process that is repeatedly executed while the control unit 11 is operating.

[0146] When the data collection process is executed, the CPU 21 of the control unit 11 executes a rule-based determination in S810 as shown in Fig. 12. In the rule-based determination, the CPU 21 makes a determination based on a numerical value, such as whether or not the value detected by the sensor is equal to or greater than a threshold value.

[0147] At S820, the CPU 21 executes an ambiguous condition determination. In the ambiguous condition determination, the CPU 21 determines a condition that is difficult to express numerically, such as "heavy rain" or "light rain," using an AI application based on an image captured by an in-vehicle camera.

[0148] At S830, the CPU 21 determines whether the collection start condition included in the collection conditions is satisfied based on the determination results of S810 and S820. The collection conditions include a collection data item indicating the type of data to be collected, a collection start condition for starting data collection, a data sampling condition, a video data trimming condition, an upload start condition indicating the timing of uploading data, etc.

[0149] The types of data to be collected include, for example, in-vehicle data such as camera data, sensor data, CAN data, and AI logic operation logs.

[0150] The conditions for starting data collection are set by, for example, an overwrite operation by the driver, a rule base based on the value of CAN data, the operation result of an application, a specific scene, or a combination of these conditions.

[0151] Driver override operations are manual interventions into functions that are automatically controlled by the vehicle system, such as the driver stepping on the brake pedal during automated driving, the driver operating the steering wheel during automated driving, switching to manual operation while the wipers are in automatic mode, canceling music shuffle playback, canceling the voice recognition function while in use, or canceling suggestions from the in-vehicle agent.

[0152] The rule base based on the value of the CAN data is, for example, when the illuminance is equal to or lower than a predetermined value, when the acceleration is equal to or higher than a predetermined value, etc. The specific scene is, for example, daytime, nighttime, rain, fog, snow, etc.

[0153] If the collection condition is not met, the CPU 21 proceeds to S870. On the other hand, if the collection condition is met, the CPU 21 collects in-vehicle data based on the collection condition in S840.

[0154] In S850, the CPU 21 converts the collected data into an upload format.

[0155] In S860, the CPU 21 transmits the data converted into the upload format to the uploader, and then proceeds to S870.

[0156] When the process proceeds to S870, the CPU 21 determines whether the upload start condition included in the collection conditions is met. If the upload start condition is not met, the CPU 21 ends the data collection process. On the other hand, if the upload start condition is met, the CPU 21 uploads the collected data corresponding to the collection conditions for which the upload start condition is met to the center 3 using the uploader, along with the collection condition ID corresponding to the collection condition, in S880, and ends the data collection process. Note that the collected data uploaded to the center 3 is data converted into the upload format in S850.

[0157] The data collection system 1 configured in this manner includes a plurality of data collection devices 2 and a center 3 .

[0158] The plurality of data collection devices 2 are mounted on a plurality of vehicles, respectively, and are configured to transmit vehicle data including at least information relating to the vehicles in which they are mounted. The center 3 is configured to receive the vehicle data from the plurality of data collection devices 2.

[0159] The center 3 stores the vehicle data received from the plurality of data collection devices 2 .

[0160] Center 3 uses the stored vehicle data to perform machine learning including annotation, model training, and model evaluation on the machine learning model, and based on the results of model evaluation obtained, identifies scenes in which the machine learning model does not meet pre-set evaluation criteria.For the identified scenes, center 3 generates collection condition data indicating collection conditions including collection data items indicating the type of vehicle data to be collected and collection start conditions for starting data collection, and stores the generated collection condition data in association with a scene tag that identifies the scene.

[0161] The center 3 distributes the collection condition data to a plurality of data collection devices 2 .

[0162] The center 3 stores the vehicle data uploaded from the plurality of data collection devices 2 based on the collection condition data, by linking the vehicle data with the scene tag.

[0163] Such a center 3 can collect vehicle data by narrowing down the data collection by scene, and can efficiently collect data from multiple data collection devices 2. Furthermore, the center 3 can efficiently generate collection condition data by identifying scenes that do not satisfy the evaluation criteria.

[0164] The center 3 generates traceability information for making it possible to trace the process of generating the collection condition data or the process of collecting the vehicle data.

[0165] The traceability information includes a collection condition profile that can link the generated collection condition data to a scene and collected data metadata D11 that can link the vehicle data to a scene, allowing the center 3 to trace the process of generating the collection condition data or the process of collecting the vehicle data.

[0166] The traceability information includes annotation job data D12 and model training job data D13 that can link the machine learning model with the vehicle data used in the machine learning for the machine learning model. This allows the center 3 to trace the process of generating the collection condition data or the process of collecting the vehicle data.

[0167] The center 3 sets access authority to the center 3 for each of the multiple functions (i.e., the collected data management and analysis unit 103, the machine learning unit 104, the collection condition management unit 105, and the CI / CD unit 106) that make up the center 3. This enables the center 3 to improve the security of the data stored in the center 3.

[0168] The center 3 sets access authority to the center 3 for each of a plurality of applications that use the machine learning model. This enables the center 3 to further improve the security of the data stored in the center 3.

[0169] In the embodiment described above, the data collection device 2 corresponds to an in-vehicle device, S40 and S50 correspond to processing as a data storage unit, S510 and S520 correspond to processing as a collection condition generation unit, and S670 corresponds to processing as a distribution unit.

[0170] Furthermore, S840 corresponds to processing performed by the collection unit, S880 corresponds to processing performed by the upload unit, and the scene tag corresponds to scene identification information.

[0171] Second Embodiment A second embodiment of the present disclosure will be described below with reference to the drawings. In the second embodiment, differences from the first embodiment will be described. The same reference numerals will be used to designate common components.

[0172] As shown in FIG. 13, the data collection system 1 of the second embodiment differs from the first embodiment in that it includes an ADASECU 200 and a cockpit ECU 300 instead of the data collection device 2.

[0173] The ADASECU 200 includes a data synchronization unit 211 , a mass production application 212 , a shadow mode application 213 , a shadow mode management unit 214 , and a scene determination AI application 215 .

[0174] The data synchronization unit 211 synchronizes the detection data detected by the sensors 201 mounted on the vehicle and the image data captured by the camera 202 mounted on the vehicle, and outputs the detection data from the sensors 201 to the mass production application 212 and outputs the image data captured by the camera 202 to the shadow mode application 213.

[0175] The production application 212 is an application that is actually applied to a vehicle.

[0176] The shadow mode application 213 is an application that performs predetermined calculations based on input data, but does not actually apply the calculation results to the vehicle.

[0177] The shadow mode management unit 214 includes an application operation data acquisition unit 221 , an operation result comparison unit 222 , a collection condition determination unit 223 , and an event notification unit 224 .

[0178] The application operation data acquisition unit 221 acquires data indicating the operation of the mass-produced application 212 and data indicating the operation of the shadow mode application 213 .

[0179] The operation result comparison unit 222 compares the operation of the mass-production application 212 with the operation of the shadow mode application 213 .

[0180] The collection condition determination unit 223 determines whether the collection condition is met based on the operation comparison result by the operation result comparison unit 222 and the determination result by the scene determination AI application 215. The collection condition is that the operation comparison results by the operation result comparison unit 222 do not match. The scene determination AI application 215 determines the scene using AI logic when a scene included in the collection condition is ambiguously specified. An ambiguously specified scene is, for example, backlit or evening.

[0181] When the collection condition determination unit 223 determines that the collection condition is met, the event notification unit 224 transmits the data collected based on the collection condition to the cockpit ECU 300 .

[0182] Cockpit ECU 300 includes an OTA unit 311, a shadow mode uploader 312, a CAN uploader 313, and a camera uploader 314. OTA is an abbreviation for Over the Air.

[0183] The OTA unit 311 includes a collection condition installation unit 321 , a vehicle state determination unit 322 , a shadow mode application installation unit 323 , and an installation possibility determination unit 324 .

[0184] The collection condition installation unit 321 installs the collection condition data distributed from the center 3 into the ADASECU 200 .

[0185] The vehicle state determination unit 322 determines the state of the vehicle in which the cockpit ECU 300 is mounted (for example, the shift position, the power supply state).

[0186] The shadow mode application installation unit 323 installs the shadow mode application distributed from the center 3 into the ADASECU 200 .

[0187] The installation possibility determination unit 324 determines whether the vehicle in which the cockpit ECU 300 is mounted is in a state in which collection condition data or a shadow mode application can be installed.

[0188] Shadow mode uploader 312 uploads the operation comparison result received from ADASECU 200 to center 3 when a preset shadow mode upload condition is met.

[0189] The CAN uploader 313 uploads the CAN data received from the ADASECU 200 to the center 3 when a preset CAN upload condition is met.

[0190] The camera uploader 314 uploads the camera data received from the ADASECU 200 to the center 3 when a preset camera upload condition is met.

[0191] Using the various data uploaded to the center 3, an application is developed that replaces the sensors 201 with cameras 202 and provides control equal to or better than conventional control.

[0192] In the data collection system 1 configured in this manner, the collection start conditions include identification information indicating a scene (for example, "backlit" or "evening" in this embodiment), and the ADASECU 200 is equipped with a scene determination AI application 215 that determines the scene specified by the identification information using data handled by the vehicle. This allows the data collection system 1 to include ambiguous expressions indicating a scene in the collection conditions. Note that the identification information may be the name of the scene, an ID indicating the scene, or a tag indicating the scene.

[0193] In the embodiment described above, the ADASECU 200 and the cockpit ECU 300 correspond to the in-vehicle device, and the data synchronization unit 211, the mass production application 212, the shadow mode application 213, the shadow mode management unit 214 and the scene determination AI application 215 correspond to the collection unit.

[0194] Furthermore, the shadow mode uploader 312, the CAN uploader 313, and the camera uploader 314 correspond to the upload unit, the mass production application 212 corresponds to the vehicle application, and the scene determination AI application 215 corresponds to the condition determination application.

[0195] Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above embodiment and can be implemented in various modifications.

[0196] [Modification 1] In the above embodiment, the collected data is uploaded to the center 3 when the upload condition is met. However, the center 3 may be configured to acquire vehicle data stored in the data collection device 2 or the cockpit ECU 300 from the data collection device 2 or the cockpit ECU 300 when a preset acquisition condition is met. This allows the center 3 to acquire vehicle data at a timing desired by the center 3.

[0197] In the above embodiment, vehicle data is collected when the comparison result between the operation of mass-production application 212 and the operation of shadow mode application 213 does not match. However, when the driver of the vehicle performs an overwrite operation in ADASECU 200, the ADASECU 200 may collect data on the driver's driving operation of the vehicle, data handled by mass-production application 212, and data handled by shadow mode application 213.

[0198] [Variation 3] In the above embodiment, the vehicle to be the distribution target is set using data for each vehicle, such as the target vehicle model, target region, and frequency of function use. However, the collection condition data may be distributed by selecting the target region, vehicle user attributes, and vehicle model for distribution of the collection condition data based on scenes identified as not meeting the evaluation criteria by machine learning model evaluation or the contents of the collection conditions. Examples of vehicle user attributes include age. This allows the center 3 to identify scenes that do not meet the evaluation criteria and distribute the collection condition data, thereby efficiently collecting vehicle data.

[0199] [Variation 4] In the above embodiment, the collection conditions for collecting scenes that do not satisfy the evaluation criteria and the collection conditions for collecting data to overcome the weaknesses of the machine learning model are generated. However, in order to collect data that is insufficient for machine learning of the machine learning model, the number of target vehicles may be increased, scenes to be collected may be added, the collection conditions may be modified, or the collected data items may be modified. In this way, the center 3 can identify the insufficient data and generate collection condition data, thereby efficiently collecting vehicle data.

[0200] [Modification 5] In the above embodiment, one collection condition corresponds to one scene, but multiple collection conditions may correspond to one scene.

[0201] For example, a first collecting condition whose collecting condition ID is "0100001" and a second collecting condition whose collecting condition ID is "0100010" may correspond to a first scene whose scene ID is "0100", and a third collecting condition whose collecting condition ID is "01110001" may correspond to a second scene whose scene ID is "0111". The scene ID corresponds to scene identification information.

[0202] The control unit 11, 31 and its method described herein may be implemented by a special-purpose computer configured by configuring a processor and memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit 11, 31 and its method described herein may be implemented by a special-purpose computer configured by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit 11, 31 and its method described herein may be implemented by one or more special-purpose computers configured by combining a processor and memory programmed to execute one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible recording medium. The method for implementing the functions of each unit included in the control unit 11, 31 does not necessarily need to include software; all of the functions may be implemented using one or more hardware components.

[0203] In the above embodiments, multiple functions of one component may be realized by multiple components, or one function of one component may be realized by multiple components. Furthermore, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Furthermore, part of the configuration of the above embodiments may be omitted. Furthermore, at least part of the configuration of the above embodiments may be added to or substituted for the configuration of another of the above embodiments.

[0204] In addition to the data collection device 2 and center 3 described above, the present disclosure can also be realized in various forms, such as a system having the data collection device 2 and center 3 as components, a program for causing a computer to function as the data collection device 2 and center 3, a non-transient physical recording medium such as a semiconductor memory on which this program is recorded, and a data collection method. [Technical Ideas Disclosed in the Present Specification] [Item 1] A data collection method executed in a center (3) configured to receive vehicle data including at least information about the vehicles from a plurality of on-board devices (2, 200, 300) mounted on each of a plurality of vehicles, the data collection method comprising: storing the vehicle data received from the plurality of on-board devices; using the stored vehicle data to perform machine learning including annotation, model training, and model evaluation on a machine learning model, identifying scenes in which the machine learning model does not satisfy a preset evaluation criterion based on a result of the model evaluation obtained; generating collection condition data for the identified scenes indicating collection conditions including collection data items indicating the type of vehicle data to be collected and collection start conditions for starting data collection; storing the generated collection condition data in association with scene identification information that identifies the scene; distributing the collection condition data to the plurality of on-board devices; and storing the vehicle data uploaded from the plurality of on-board devices based on the collection condition data in association with the scene identification information.

[0205] [Item 2] The data collection method according to Item 1, further comprising generating traceability information for enabling tracing of the process of generating the collection condition data or the process of collecting the vehicle data.

[0206] [Item 3] The data collection method according to Item 2, wherein the traceability information includes information that can link the generated collection condition data to the scene and information that can link the vehicle data to the scene.

[0207] [Item 4] The data collection method according to Item 2, wherein the traceability information includes information that can link the machine learning model with the vehicle data used in the machine learning for the machine learning model.

[0208] [Item 5] The data collection method according to any one of items 1 to 4, wherein access authority to the center is set for each of a plurality of functions that constitute the center.

[0209] [Item 6] The data collection method according to any one of items 1 to 5, wherein access authority to the center is set for each of a plurality of applications that use the machine learning model.

[0210] [Item 7] The data collection method according to any one of items 1 to 6, wherein, based on the results of the model evaluation or the contents of the collection conditions, a region, an attribute of a vehicle user, and a vehicle type to which the collection condition data is to be distributed are selected, and the collection condition data is distributed.

[0211] [Item 8] The data collection method according to any one of items 1 to 7, wherein, based on the results of the model evaluation, the data collection method increases the number of target vehicles, adds the scenes to be collected, modifies the collection conditions, or modifies the collected data items in order to collect data that is insufficient in the machine learning for the machine learning model.

[0212] [Item 9] The data collection method according to any one of items 1 to 8, wherein the collection start condition includes identification information that indicates the scene, and the plurality of in-vehicle devices are equipped with a condition determination application that determines the scene specified by the identification information using data handled in the vehicle.

[0213] [Item 10] A data collection method according to any one of items 1 to 9, wherein the collection start condition includes an overwrite operation being performed by a driver of the vehicle, and the collection data items include driving operation data of the vehicle performed by the driver, data handled by a vehicle-applied application (212) that is mounted on the vehicle and applies control related to the overwrite operation to the vehicle, and data handled by a shadow mode application (213) that is mounted on the vehicle and operates in a shadow mode.

[0214] [Item 11] A center (3) configured to receive vehicle data including at least information about the vehicles from a plurality of on-board devices (2, 200, 300) mounted on each of a plurality of vehicles, the center (3) comprising: a data storage unit (S40, S50) configured to store the vehicle data received from the plurality of on-board devices; a collection condition generation unit (S510, S520) configured to: use the stored vehicle data to perform machine learning including annotation, model training, and model evaluation on a machine learning model, to identify a scene in which the machine learning model does not satisfy a preset evaluation criterion based on a result of the model evaluation obtained by the execution of the machine learning, including annotation, model training, and model evaluation, on the machine learning model; generate collection condition data for the identified scene indicating collection conditions including collection data items indicating a type of the vehicle data to be collected and a collection start condition for starting data collection; and store the generated collection condition data in association with scene identification information that identifies the scene; and a distribution unit (S670) configured to distribute the collection condition data to the plurality of on-board devices. The data storage unit is configured to store the vehicle data uploaded from the plurality of in-vehicle devices based on the collection condition data in association with the scene identification information.

[0215] [Item 12] A computer of a center (3) configured to receive vehicle data including at least information about the vehicles from a plurality of on-board devices (2, 200, 300) mounted on each of a plurality of vehicles is configured to function as: a data storage unit (S40, S50) configured to store the vehicle data received from the plurality of on-board devices; a collection condition generation unit (S510, S520) configured to identify a scene in which the machine learning model does not satisfy a preset evaluation standard based on a result of the model evaluation obtained by performing machine learning including annotation, model training, and model evaluation on a machine learning model using the stored vehicle data, generate collection condition data for the identified scene indicating collection conditions including collection data items indicating the type of vehicle data to be collected and collection start conditions for starting data collection, and store the generated collection condition data in association with scene identification information that identifies the scene; and a distribution unit (S670) configured to distribute the collection condition data to the plurality of on-board devices. The data storage unit stores the vehicle data uploaded from the plurality of in-vehicle devices based on the collection condition data in association with the scene identification information.

[0216] [Item 13] A data collection system (1) comprising: a plurality of on-board devices (2, 200, 300) mounted on each of a plurality of vehicles and configured to transmit vehicle data including at least information about the mounted vehicles; and a center (3) configured to receive the vehicle data from the plurality of on-board devices, wherein the center includes: a data storage unit (S40, S50) configured to store the vehicle data received from the plurality of on-board devices; and a collection condition generation unit (S510, S520) configured to: identify a scene in which the machine learning model does not satisfy a preset evaluation standard based on a result of the model evaluation obtained by performing machine learning including annotation, model training, and model evaluation on a machine learning model using the stored vehicle data; generate collection condition data for the identified scene indicating collection conditions including collection data items indicating a type of the vehicle data to be collected and a collection start condition for starting data collection; and store the generated collection condition data in association with scene identification information that identifies the scene; a distribution unit (S670) configured to distribute the collection condition data to the plurality of on-board devices, wherein the data storage unit is configured to store the vehicle data uploaded from the plurality of on-board devices based on the collection condition data in association with the scene identification information, and the plurality of on-board devices are configured to collect the vehicle data corresponding to the collection data items based on the collection conditions indicated by the collection condition data when the collection start condition is met, upon receiving the collection condition data from the center, and an upload unit (S880, 312 to S314) configured to upload the vehicle data collected based on the collection conditions to the center in association with the collection conditions.

[0217] [Item 14] A data collection method executed in a data collection system (1) including a plurality of on-board devices (2, 200, 300) mounted on each of a plurality of vehicles and configured to transmit vehicle data including at least information about the mounted vehicles, and a center (3) configured to receive the vehicle data from the plurality of on-board devices, wherein the center: stores the vehicle data received from the plurality of on-board devices; uses the stored vehicle data to perform machine learning including annotation, model training, and model evaluation on a machine learning model, and based on the result of the model evaluation obtained, identifies a scene in which the machine learning model does not satisfy a preset evaluation criterion; generates collection condition data for the identified scene indicating collection conditions including collection data items indicating a type of the vehicle data to be collected and a collection start condition for starting data collection; stores the generated collection condition data in association with scene identification information that identifies the scene; distributes the collection condition data to the plurality of on-board devices; and stores the vehicle data uploaded from the plurality of on-board devices based on the collection condition data in association with the scene identification information; and When the collection condition data is received from the center, the vehicle data corresponding to the collection data item is collected based on the collection conditions indicated by the collection condition data at the timing when the collection start condition is met, and the vehicle data collected based on the collection conditions is linked to the collection conditions and uploaded to the center.

Claims

1. A data collection method executed by a center (3) configured to receive vehicle data including at least information on the vehicle on which it is mounted from a plurality of in-vehicle devices (2, 200, 300) mounted on each of a plurality of vehicles, the method comprising: storing the vehicle data received from the plurality of in-vehicle devices; using the stored vehicle data, based on the result of model evaluation obtained by performing machine learning including annotation, model training, and model evaluation on a machine learning model, identifying a scene in which the machine learning model does not meet a preset evaluation criterion; for the identified scene, generating collection condition data indicating a collection condition including a collection data item indicating the type of vehicle data to be collected and a collection start condition for starting data collection; storing the generated collection condition data in association with scene identification information for identifying the scene; distributing the collection condition data to the plurality of in-vehicle devices; and storing the vehicle data uploaded from the plurality of in-vehicle devices based on the collection condition data in association with the scene identification information.

2. The data collection method according to claim 1, the method generating traceability information for making the process of generating the collection condition data or the process of collecting the vehicle data traceable.

3. The data collection method according to claim 2, wherein the traceability information includes information capable of associating the generated collection condition data with the scene and information capable of associating the vehicle data with the scene.

4. The data collection method according to claim 2, wherein the traceability information includes information capable of associating the machine learning model with the vehicle data used in the machine learning for the machine learning model.

5. The data collection method according to any one of claims 1 to 4, the method setting access rights to the center for each of a plurality of functions constituting the center.

6. The data collection method according to any one of claims 1 to 4, the method setting access rights to the center for each of a plurality of applications using the machine learning model.

7. The data collection method according to any one of claims 1 to 4, wherein a region, an attribute of a vehicle user, and a vehicle type to which the collection condition data is to be distributed are selected based on the result of the model evaluation or the content of the collection condition, and the collection condition data is distributed.

8. The data collection method according to any one of claims 1 to 4, wherein, based on the result of the model evaluation, in order to collect data that is insufficient in the machine learning for the machine learning model, an increase in the target vehicle, an addition of the scene to be collected, a modification of the collection condition, or a modification of the collection data item is performed.

9. The data collection method according to any one of claims 1 to 4, wherein the collection start condition includes identification information indicating the scene, and a condition determination application for determining the scene specified by the identification information using data handled by the vehicle is installed in a plurality of the in-vehicle devices.

10. The data collection method according to any one of claims 1 to 4, wherein the collection start condition includes an overwrite operation being performed by a driver of the vehicle, and the collection data items include driving operation data of the vehicle by the driver, data handled by a vehicle application (212) mounted on the vehicle and applying control related to the overwrite operation to the vehicle, and data handled by a shadow mode application (213) mounted on the vehicle and operating in a shadow mode.

11. A center (3) configured to receive vehicle data at least including information on the vehicle on which it is mounted from a plurality of in-vehicle devices (2, 200, 300) mounted on each of a plurality of vehicles, the center comprising: a data storage unit (S40, S50) configured to store the vehicle data received from the plurality of in-vehicle devices; based on the result of the model evaluation obtained by performing machine learning including annotation, model training, and model evaluation on a machine learning model using the stored vehicle data, identifying a scene in which the machine learning model does not meet a preset evaluation criterion, and for the identified scene, generating collection condition data indicating a collection condition including a collection data item indicating the type of vehicle data to be collected and a collection start condition for starting data collection, and storing the generated collection condition data in association with scene identification information for identifying the scene; a collection condition generation unit (S510, S520) configured to generate the collection condition data; and a distribution unit (S670) configured to distribute the collection condition data to the plurality of in-vehicle devices, wherein the data storage unit is configured to store the vehicle data uploaded from the plurality of in-vehicle devices based on the collection condition data in association with the scene identification information.

12. A computer of a center (3) configured to receive vehicle data including at least information on the vehicle on which each of a plurality of in-vehicle devices (2, 200, 300) mounted on a plurality of vehicles is mounted, a data storage unit (S40, S50) configured to store the vehicle data received from the plurality of in-vehicle devices, and based on the result of the model evaluation obtained by performing machine learning including annotation, model training, and model evaluation on a machine learning model using the stored vehicle data, identify a scene in which the machine learning model does not meet a preset evaluation criterion, and for the identified scene, generate collection condition data indicating a collection condition including a collection data item indicating the type of vehicle data to be collected and a collection start condition for starting data collection, and store the generated collection condition data in association with scene identification information for identifying the scene, and a collection condition generation unit (S510, S520) configured to function as a distribution unit (S670) configured to distribute the collection condition data to the plurality of in-vehicle devices, and the data storage unit is configured to store the vehicle data uploaded from the plurality of in-vehicle devices based on the collection condition data in association with the scene identification information, a data collection program.

13. A data collection system (1) comprising: a plurality of in-vehicle devices (2, 200, 300) mounted on each of a plurality of vehicles and configured to transmit vehicle data including at least information regarding the vehicle on which they are mounted; and a center (3) configured to receive the vehicle data from the plurality of in-vehicle devices. The center includes: a data storage unit (S40, S50) configured to store the vehicle data received from the plurality of in-vehicle devices; a collection condition generation unit (S510, S520) configured to identify a scene in which the machine learning model does not meet a preset evaluation criterion based on the result of the model evaluation obtained by performing machine learning including annotation, model training, and model evaluation on the machine learning model using the stored vehicle data, and generate collection condition data indicating a collection condition including a collection data item indicating the type of vehicle data to be collected and a collection start condition for starting data collection for the identified scene, and store the generated collection condition data in association with scene identification information for identifying the scene; and a distribution unit (S670) configured to distribute the collection condition data to the plurality of in-vehicle devices. The data storage unit is configured to store the vehicle data uploaded from the plurality of in-vehicle devices based on the collection condition data in association with the scene identification information. The plurality of in-vehicle devices include: a collection unit (S840, 211 to 215) configured to collect the vehicle data corresponding to the collection data item at the timing when the collection start condition is satisfied based on the collection condition indicated by the collection condition data when the collection condition data is received from the center; and an upload unit (S880, 312 to S314) configured to upload the vehicle data collected based on the collection condition to the center in association with the collection condition.

14. A data collection method executed in a data collection system (1) comprising a plurality of in-vehicle devices (2, 200, 300) configured to be mounted on each of a plurality of vehicles and transmit vehicle data including at least information regarding the vehicle on which they are mounted, and a center (3) configured to receive the vehicle data from the plurality of in-vehicle devices, wherein the center stores the vehicle data received from the plurality of in-vehicle devices, and based on the result of the model evaluation obtained by performing machine learning including annotation, model training, and model evaluation on a machine learning model using the stored vehicle data, identifies a scene in which the machine learning model does not meet a preset evaluation criterion, and for the identified scene, generates collection condition data indicating a collection condition including a collection data item indicating the type of vehicle data to be collected and a collection start condition for starting data collection, stores the generated collection condition data in association with scene identification information for identifying the scene, distributes the collection condition data to the plurality of in-vehicle devices, stores the vehicle data uploaded from the plurality of in-vehicle devices based on the collection condition data in association with the scene identification information, and the plurality of in-vehicle devices, when receiving the collection condition data from the center, collect the vehicle data corresponding to the collection data item at the timing when the collection start condition is satisfied based on the collection condition indicated by the collection condition data, and upload the vehicle data collected based on the collection condition to the center in association with the collection condition.

Images