Vehicle data collection system, vehicle data management device, and vehicle data collection device

Abstract

This vehicle data collection system comprises a CPU (10a, 20a) and collects data pertaining to a vehicle (1) in a server (20). The CPU (10a, 20a) is configured to execute: extracting, from among events occurring in the vehicle (1) or an environment in which the vehicle (1) is located, an event for causing a user of the vehicle 1 to perform a usefulness-increasing operation that enhances the usefulness of the data; and causing a CID (12) and / or a smartphone (40) to provide a user interface for performing the usefulness-increasing operation for data related to the extracted event.

Description

Vehicle data collection system, vehicle data management device, and vehicle data collection device CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Patent Application No. 2024-118811 filed in Japan on July 24, 2024, the contents of which are incorporated by reference in their entirety.

[0002] The disclosure of this specification relates to a technology for collecting vehicle data at a server.

[0003] Patent Document 1 discloses that a computing device in a server collects vehicle data from the vehicle in accordance with a data collection policy included in the computing device.

[0004] US Patent Application Publication No. 2022 / 0204010

[0005] However, when data collection is performed solely by relying on mechanical processing such as the data collection policy described in Patent Document 1, there is a possibility that a large amount of unnecessary data will be collected by the server, and there is a concern that the cost of processing such a large amount of unnecessary data on the server side will increase.

[0006] One of the purposes of the disclosure of this specification is to provide a vehicle data collection system, a vehicle data management device, and a vehicle data collection device that are capable of efficiently collecting useful data.

[0007] One aspect disclosed herein is a vehicle data collection system that includes at least one processing unit and collects vehicle data on a server, wherein the at least one processing unit is configured to: extract events that occur in the vehicle or the environment in which the vehicle is located, and allow the user of the vehicle to perform usefulness operations to increase the usefulness of the data; and cause at least one device to provide a user interface that allows the user to perform usefulness operations on data related to the extracted events.

[0008] Another disclosed aspect is a vehicle data management device that has at least one processing unit, is mounted on a vehicle that is communicatively connected to a server, and is configured to upload vehicle data collected by the server from the vehicle to the server, wherein the at least one processing unit is configured to: extract events that occur in the vehicle or the environment in which the vehicle is located, and allow the vehicle user to perform usefulness operations to increase the usefulness of the data; and cause the at least one device to provide a user interface that allows the user to perform usefulness operations on data related to the extracted events.

[0009] Another disclosed aspect is a vehicle data collection device that has at least one processing unit, is communicatively connected to a vehicle, and is configured to collect vehicle data, wherein the at least one processing unit is configured to: acquire data and extract events that occur in the vehicle or the environment in which the vehicle is located, and that allow a user of the vehicle to perform a usability improvement operation to increase the usability of the data; and cause the at least one device to provide a user interface that allows a user to perform a usability improvement operation on data related to the extracted events.

[0010] According to these aspects, vehicle data can be made useful as a result of user operations realized by providing a user interface. In other words, data accuracy can be improved without relying solely on mechanical processing. Therefore, useful data can be efficiently collected at the server.

[0011] Note that the symbols in parentheses included in the claims etc. are intended to exemplify the correspondence with the parts of the embodiments described below, and are not intended to limit the technical scope.

[0012] 1 is a diagram schematically showing the hardware configuration of a vehicle data collection system; 2 is a diagram schematically showing the functional configuration of a vehicle data collection system; 3 is a flowchart illustrating an example of overall processing related to a data collection function; 4 is a flowchart illustrating processing related to annotation of vehicle data; 5 is a diagram schematically showing the hardware configuration of a vehicle data collection system, etc.; 6 is a flowchart illustrating a process for determining the timing of annotation; 7 is a diagram schematically showing the hardware configuration of a vehicle data collection system; 8 is a diagram illustrating an example of a user interface; 9 is a diagram illustrating an example of a user interface; 10 is a diagram illustrating an example of a user interface; 11 is a diagram illustrating an example of a user interface; 12 is a flowchart illustrating an example of overall processing related to a data collection function; 13 is a diagram schematically showing the functional configuration of a vehicle data collection system;

[0013] In this disclosure and claims, the term "processor" refers to one or more hardware processors configured to load computer program code (i.e., one or more instructions of a computer program) included in a computer program and execute the processing defined by the code. In other words, a "processor" is a hardware device that executes one or more programmed processes. Therefore, computer program code can also be considered software that can define the processing of the processor depending on its content. For example, a "processor" may be a general-purpose or special-purpose processor, such as, but not limited to, a CPU, a microprocessor, a GPU, and a DFP (Data Flow Processor).

[0014] In this disclosure and in the claims, the term "memory" refers to one or more hardware memories that are non-transitory tangible recording media and configured to store computer program code and / or data accessible to a processor. The "memory" may be implemented using memory technologies such as SRAM, SDRAM, non-volatile flash memory, or other types of memory. Computer program code constituting a program may be stored in the memory and executed by a processor to cause the processor to perform the various functions described above.

[0015] In this disclosure and in the claims, the term "circuit" refers to one or more hardware logic circuits configured to perform specific processing based on a pre-designed circuit configuration. In other words (and in contrast to "processor"), a "circuit" in this disclosure and in the claims refers to a hardware device that performs specific processing based on a circuit configuration, rather than a software-based process such as computer program code. For example, a "circuit" may include custom ICs such as application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs) designed using a hardware description language (HDL). In other words, a "circuit" in this disclosure and in the claims includes all hardware circuits except for a processor that executes processing by loading computer program code.

[0016] In this disclosure or in the claims, the phrase "at least one circuit and processor" should be construed as a disjunction (a logical OR), and not as at least one circuit and at least one processor.

[0017] In this disclosure or claims, the term "processing unit" refers to a hardware device that performs processing using a "processor," a "circuit," or a combination thereof. The "processing unit" may refer to a "processor" itself if its function is interpreted as being not achievable by a "circuit" but is achievable by a "processor."

[0018] Hereinafter, several embodiments will be described with reference to the drawings. Note that corresponding components in each embodiment are given the same reference numerals, and redundant description may be omitted. When only a portion of the configuration is described in each embodiment, the configuration of another embodiment described previously can be applied to the remaining portion of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of several embodiments can also be partially combined together even if not explicitly stated, as long as there is no particular problem with the combination.

[0019] (First Embodiment) A vehicle data collection system DCS of a first embodiment shown in Fig. 1 collects data from a vehicle 1. The collected data is used, for example, to develop the hardware of the vehicle 1 or applications used in the vehicle 1. The collected data may be used for urban planning, such as improving traffic signal control and road infrastructure. The collected data may also be used for planning the opening of commercial facilities, etc.

[0020] Vehicle 1 is a vehicle that can participate in public road traffic. Vehicle 1 may be, for example, a vehicle that can be manually driven by a driver, such as a four-wheeled automobile or truck. Vehicle 1 may also be a vehicle that can perform automated driving. The automation level of vehicle 1 is classified into levels 1 to 5, for example, as defined in SAE J3016. Vehicle 1 may be capable of achieving any of the levels of driving.

[0021] At levels 0 to 2, the driver performs some or all of the dynamic driving tasks. Levels 0 to 2 may be classified as so-called manual driving. Level 0 indicates that driving is not automated. Level 1 indicates that the driver is assisted. Level 2 indicates that driving is partially automated. Levels 3 to 5 may be classified as so-called automated driving. Level 3 indicates that driving is conditionally automated. Level 4 indicates that driving is highly automated. Level 5 indicates that driving is fully automated.

[0022] The vehicle data collection system DCS includes a data management device 10 and a server 20. The vehicle data collection system DCS may further include a development computer 30. Although the following description focuses on one vehicle 1 and its data management device 10, the vehicle data collection system DCS may also be configured to collect data from multiple vehicles 1. In other words, the vehicle data collection system DCS may include multiple data management devices 10 that individually correspond to each vehicle 1.

[0023] The vehicle data collection system DCS is suitable for collecting data (hereinafter referred to as vehicle data) of vehicles 1 in long-tail cases encountered by vehicles 1 that can participate in public road traffic. Long-tail cases are cases that cannot be covered by development in a closed environment. Collecting vehicle data in long-tail cases can improve the coverage of applications and their functions in real-world environments.

[0024] The data management device 10 is an in-vehicle device that is installed in the vehicle 1 and manages vehicle data. The data management device 10 acquires vehicle data from the vehicle 1 and executes processing for transferring the vehicle data to the server 20.

[0025] The data management device 10 may be, for example, an ECU (Electronic Control Unit) mainly composed of a computer. The computer constituting the data management device 10 has at least one CPU (Central Processing Unit) 10a and one memory 10c. The memory 10c non-temporarily stores computer programs and data that can be read by the CPU 10a. The computer may also be provided with a rewritable volatile storage medium such as a RAM (Random Access Memory) 10d. The CPU 10a can execute various processes in accordance with the computer programs stored in the memory 10c. The computer also includes an interface 10b for exchanging data with the outside.

[0026] The data management device 10 also includes a storage 10e for recording vehicle data. The storage 10e is a rewritable nonvolatile storage medium such as a semiconductor memory, a magnetic medium, or an optical medium. The storage 10e has a storage area with a finite storage capacity.

[0027] The vehicle data is data generated by the vehicle 1. The vehicle data may be sensor data generated by a sensor mounted on the vehicle 1 detecting the external or internal environment of the vehicle 1. The vehicle data may be data related to a diagnostic code generated to warn of at least one of a failure and an abnormality in the vehicle 1.

[0028] The vehicle data may be data generated during or as a result of an operation performed by an application running on the vehicle 1. The application referred to here may be a software unit that realizes a specific function and is operated by one or more pieces of software. Such an application may be operated by the CPU 10a executing a computer program stored in the memory 10c in the data management device 10. Alternatively, the application may be operated by a CPU in a device installed in the vehicle, separate from the data management device 10, executing a computer program stored in the memory.

[0029] The data management device 10 is configured to transmit the vehicle data stored in the storage 10e to the server 20 using a DCM (Data Communication Module) 11. The DCM 11 is a communication module mounted on the vehicle 1. The DCM 11 transmits and receives radio waves to and from base stations around the vehicle 1 through wireless communication conforming to communication standards such as LTE (Long Term Evolution) and 5G. By mounting the DCM 11, the vehicle 1 becomes a connected car that can connect to the Internet. Furthermore, the DCM 11 cooperates with the data management device 10 to transmit the vehicle data to the server 20.

[0030] The server 20 is a vehicle data collection device that is installed in an external environment of the vehicle 1. The server 20 is configured to be able to collect the vehicle data generated by the data management device 10 by being connected to, for example, the Internet.

[0031] The server 20 may be configured by a single processing device or may be a cloud server. A cloud server is a server on a network realized by cloud computing, and is realized by multiple processing devices located in remote locations apart from each other.

[0032] The processing device in the server 20 is mainly composed of, for example, a computer. The computer constituting the processing device has at least one CPU 20a and one memory 20c. The memory 20c non-temporarily stores computer programs and data that can be read by the CPU 20a. The computer may also be provided with a rewritable volatile storage medium such as a RAM 15d.

[0033] The CPU 20a can execute various processes in accordance with computer programs stored in the memory 20c. The computer further includes an interface 20b for exchanging data with the outside.

[0034] The processing device also includes a storage 20e for recording vehicle data. The storage 20e is a rewritable non-volatile storage medium such as a semiconductor memory, a magnetic medium, or an optical medium. The storage 20e has a limited storage area with a much larger storage capacity than the storage 10e of the data management device 10. The server 20 stores the vehicle data collected from the vehicle 1 in the storage 20e.

[0035] The development computer 30 is a development device provided in the external environment of the vehicle 1. The development computer 30 may be, for example, a personal computer operated by a developer of the vehicle 1 or an application for the vehicle 1. The developer may be, for example, an engineer who is knowledgeable about vehicle control and software.

[0036] The development computer 30 has at least one CPU 30a and one memory 30c. The memory 30c non-temporarily stores computer programs and data that can be read by the CPU 30a. The computer may also be provided with a rewritable volatile storage medium such as a RAM (Random Access Memory) 30d. The CPU 30a can execute various processes in accordance with the computer programs stored in the memory 30c.

[0037] Furthermore, development computer 30 includes interface 30b for exchanging data with the outside. Interface 30b includes a communications interface for connecting development computer 30 to the Internet, and an operation interface for inputting signals and operating devices such as a keyboard and mouse for accepting operations from the developer. Development computer 30 is communicably connected to server 20, and is able to exchange data required for development by the developer.

[0038] Next, an example of a logical architecture for the vehicle data collection system DCS to realize the vehicle data collection function will be shown with reference to FIG.

[0039] The data management device 10 may include a trigger management unit V1 and a storage and transmission unit V4 as functional units whose functions are realized when the CPU 10a executes a computer program stored in the memory 10c. The data management device 10 may include at least one of a current application control unit V2 and a development application control unit V3 as functional units whose functions are realized when the CPU 10a executes a computer program stored in the memory 10c.

[0040] The server 20 may have at least one of a development application distribution unit C1, an instruction setting unit C2, and a data collection unit C3 as functional units whose functions are realized by the CPU 20a executing a computer program stored in the memory 20c.

[0041] The development computer 30 may include at least one of a scene extraction unit D1, a labeling unit D2, a machine learning unit D3, a verification unit D4, and an upload unit D5 as functional units whose functions are realized when the CPU 30a executes a computer program stored in the memory 30c.

[0042] Each functional unit will be described in detail below. The trigger management unit V1 acquires, from the instruction setting unit C2, acquisition instruction information that indicates the vehicle data to be collected in the vehicle 1, using the DCM 11. Based on the acquisition instruction information, the trigger management unit V1 sets a trigger condition that serves as a trigger for recording and transmitting the vehicle data. The trigger condition may be the condition specified by the acquisition instruction information (hereinafter referred to as the instruction condition), or may be a changed condition that has been changed by a review process of the instruction condition.

[0043] While the vehicle 1 is in use, the trigger management unit V1 successively determines whether the trigger condition is currently satisfied. As a result, if the state transitions from a state in which the trigger condition is not satisfied to a state in which the trigger condition is satisfied, the trigger management unit V1 requests the current application control unit V2 and the development application control unit V3 to start acquiring and recording vehicle data. As a result, if the state transitions from a state in which the trigger condition is satisfied to a state in which the trigger condition is not satisfied, the trigger management unit V1 requests the current application control unit V2 and the development application control unit V3 to stop acquiring and recording vehicle data. Note that the trigger condition here may basically be the same for both the current application control unit V2 and the development application control unit V3. However, different conditions may also be set.

[0044] The current application control unit V2 controls the current application, which is an application that is currently being used officially to control the vehicle 1 in the vehicle 1. The current application control unit V2 starts and operates the current application based on an operation by a driver or other occupant to enable the current application.

[0045] The current application control unit V2 also receives a request to record vehicle data from the trigger management unit V1. If a recording request has been received, the current application control unit V2 acquires the vehicle data requested to be recorded and related to the operation of the current application, and requests that the data be recorded in the storage and transmission unit V4.

[0046] The development application control unit V3 controls a development application, which is an application under development and distributed from the server 20. The development application may be an application that is an improvement of a current application having similar functions, or may be an application with more advanced functions than the current application.

[0047] The development application control unit V3 operates the development application in a special mode, such as a shadow mode, a ghost mode, etc. In other words, the development application is tested under the special mode operation.

[0048] Shadow mode is a mode in which a development application runs in parallel with the current application behind the scenes, for example, when autonomous driving is in operation. Shadow mode runs the development application so that vehicle data related to the operation of the development application can be obtained in real time without activating the actual vehicle control function of the development application. Shadow mode makes it possible to evaluate the performance and behavior of the development application while restricting the impact on actual vehicle control and the driver.

[0049] Ghost mode is a mode in which a development application runs in parallel behind the scenes while, for example, a driver is actually driving. The development application here is an application that provides an automated driving function with an automation level of 3 or higher. In ghost mode, the driver's actions and the behavior of the vehicle 1 are tracked, and information on the performance of the development application can be collected by comparing the driver's driving with the behavior of the automated driving function. In ghost mode, differences between the driver's driving and the behavior of the automated driving function are evaluated, and improvements to the development application can be identified.

[0050] If the development application is an improved version of the current application, the development application control unit V3 may start and operate the development application so as to link with the current application based on an operation by a driver or other occupant to enable the current application, etc. However, as described above, unlike the current application, the output of the development application is not reflected in the actuators and HMI (Human Machine Interface) of the vehicle 1.

[0051] The HMI of the vehicle 1 may be, for example, an information notification device 12 that notifies an occupant such as a driver in the vehicle of information, or may be an operation device that allows the driver to operate the vehicle 1. The information notification device 12 may be a meter display, a head-up display, a center information display (CID) 12, or the like.

[0052] The CID 12 is an in-vehicle display device that is disposed, for example, in the center of the instrument panel of the vehicle 1. The CID 12 includes a display 12a and a touch panel 12b.

[0053] The display 12a is, for example, a liquid crystal display or an organic EL display, and displays images on the display screen, and can present information to the user in cooperation with a speaker, etc. The touch panel 12b is configured integrally with the display 12a, and can detect a user's touch operation on the display screen and provide the detection result to an in-vehicle device such as the data management device 10.

[0054] The development application control unit V3 may monitor the operation of the development application and diagnose any abnormalities in the development application. However, because the output of the development application does not affect vehicle control, the development application control unit V3 may not forcibly terminate the development application when it detects an abnormality, and may decide not to warn the driver of the abnormality using the information notification device 12.

[0055] The development application control unit V3 also receives a request to record vehicle data from the trigger management unit V1. If a recording request has been received, the development application control unit V3 acquires the vehicle data requested to be recorded and that is related to the operation of the development application, and requests that the data be recorded in the storage and transmission unit V4.

[0056] The storage and transmission unit V4 sequentially records the requested vehicle data in the storage 10e in response to recording requests from the current application control unit V2 and the development application control unit V3. This recording may be temporary storage until the vehicle data is transmitted.

[0057] Before recording data in the storage 10e, the storage transmission unit V4 checks the remaining storage capacity of the storage 10e (hereinafter referred to as the remaining storage capacity). If the remaining storage capacity is greater than a preset threshold, the storage transmission unit V4 records the requested vehicle data. If the remaining storage capacity is equal to or less than the preset threshold, the storage transmission unit V4 deletes some or all of the data already stored in the storage 10e in order to record the requested vehicle data.

[0058] Furthermore, the storage transmission unit V4 transmits the data recorded in the storage 10e to the server 20 via the DCM 11. The data transmission may be performed at a timing that depends on the storage capacity, the communication environment of the vehicle 1, etc. The storage transmission unit V4 deletes the vehicle data that has been transmitted to the server 20 from the storage 10e.

[0059] In the server 20, the data collection unit C3 manages the vehicle data transmitted from the vehicle 1. Specifically, the data collection unit C3 sequentially stores in the storage 20e the vehicle data transmitted from the storage transmission unit V4 of the vehicle 1. When the server 20 is communicably connected to a plurality of vehicles 1 and collects vehicle data from the plurality of vehicles 1, the vehicle data transmitted from the plurality of vehicles 1 may be sequentially stored in a mixed state in the same storage 20e.

[0060] Furthermore, in response to a download request from development computer 30, data collection unit C3 transmits vehicle data stored in storage 20e to development computer 30. The vehicle data transmitted at this time may be data corresponding to the download request, or may be all or a portion of the data stored in storage 20e.

[0061] In the development computer 30, at least one of the scene extraction unit D1 and the labeling unit D2 requests a download from the server 20. When the server 20 provides vehicle data in response to the download request, the scene extraction unit D1 and the labeling unit D2 classify the large amount of vehicle data. The scene extraction unit D1 extracts, from the vehicle data, vehicle data related to scenes required for verifying the development application. The extracted scenes may be scenes specified by the developer through input operations on the interface 30b, etc. The scenes here may be replaced with scenarios represented by a sequence of scenes that define a situation at a given moment.

[0062] The labeling unit D2 labels the downloaded vehicle data. The labeling here may be adding or associating information about the scene extracted based on the processing by the scene extraction unit D1 to the corresponding vehicle data. The labeling here may also be labeling of other information.

[0063] The vehicle data classified by the scene extraction unit D1 and the labeling unit D2 in this manner is used by the machine learning unit D3. The machine learning unit D3 performs machine learning using the vehicle data. The machine learning here may be optimization of a model used in the development application and configured mainly using a neural network or the like, using the vehicle data, with a technique such as deep learning. The optimization here may be optimization of various parameters used in the development application. The development application can be further improved by the machine learning unit D3.

[0064] The verification unit D4 executes a process for verifying the improved developed application. The verification process here may be offline or online. The verification process here may be autonomous verification without the involvement of the developer, in which a conclusion is reached as to whether the developed application has any defects through processing of a verification program stored in memory 30c. The verification here may involve outputting verification data required for the developer to perform the verification through processing of the verification program stored in memory 30c.

[0065] Based on the results of the verification using the verification unit D4, the developer may decide to use the tested developed application as is, or may decide to improve the developed application and test it again. In the latter case, the developer uses the function of the upload unit D5.

[0066] The upload unit D5 uploads the development application created by the developer to the server 20. The upload unit D5 may further upload a data collection plan associated with the development application. The data collection plan may include the type of vehicle data to be collected in testing the development application and the scale of the vehicle data. The scale may include the number of vehicles from which vehicle data is to be collected, or the total number of specific vehicle data required to be acquired. The data collection plan may further include information for identifying the vehicles 1 from which data is to be collected. This information may, for example, directly and individually specify the vehicles from which data is to be collected, or may be specified by setting conditions such as vehicle type. When the vehicles from which data is to be collected are individually specified, the data collection plan may include the instruction information itself, as described below.

[0067] The data collection plan may be set by the developer through input operations into the interface 30b, or may be generated by the upload unit D5.

[0068] The development application distribution unit C1 in the server 20 identifies the vehicles 1 from which data will be collected based on the data collection plan. The development application distribution unit C1 distributes the development application received from the development computer 30 to the identified vehicles 1.

[0069] In the server 20, the instruction setting unit C2 sets collection instructions for each vehicle 1 that is a target for data collection and that is identified by the development application distribution unit C1 based on the data collection plan, and generates acquisition instruction information for sharing the instructions. The instruction setting unit C2 distributes the generated acquisition instruction information to each vehicle 1. The distribution of the development application and the acquisition instruction information may be performed at the same time or at different times.

[0070] Next, an example of a processing method for implementing the data collection function using the vehicle data collection system DCS will be described using the flowchart in Figure 3. Steps S1 to S8 in this flowchart outline the overall processing flow for the data collection function. The series of steps S1 to S8 may be implemented, for example, by the CPUs 10a, 20a, and 30a of the data management device 10, server 20, and development computer 30 executing computer programs stored in their memories 10c, 20c, and 30c.

[0071] In S1, the development computer 30 uploads the development application and data collection plan to the server 20.

[0072] In S2 after the processing of S1, the server 20 generates acquisition instruction information for each vehicle 1 based on the data collection plan. In S3 after the processing of S2, the server 20 distributes the developed application and the acquisition instruction information to each vehicle.

[0073] In S4 after processing S3, the data management device 10 in each vehicle 1 sets a trigger condition, which is a condition that triggers starting to acquire and record vehicle data, based on the acquisition instruction information. In S5 after processing S4, the data management device 10 acquires and records vehicle data based on the trigger condition. In S6 after processing S5, the data management device 10 uploads the vehicle data to the server 20.

[0074] In S7 after the processing of S6, the development computer 30 downloads the vehicle data uploaded to the server 20 and executes processing to classify the vehicle data. In S8 after the processing of S7, the development computer 30 executes processing to verify the development application based on the classified vehicle data. The series of processes ends with S8.

[0075] The following describes in more detail the process executed by the storage and transmission unit V4 to transmit vehicle data to the server 20 via the DCM 11. In this embodiment, the storage and transmission unit V4 determines to have the user of the vehicle 1 (e.g., the driver) perform a utility operation for some of the events occurring in the vehicle 1 or the environment of the vehicle 1.

[0076] The usefulness operation here refers to an operation for increasing the usefulness of vehicle data related to events occurring in the vehicle 1 or the environment of the vehicle 1. As a result of the usefulness operation, the vehicle data related to events occurring in the vehicle 1 or the environment of the vehicle 1 may be transmitted to the server 20 with additional information for usefulness added. The usefulness operation may also be referred to as annotation, and will be referred to as annotation hereinafter. In the following, among events occurring in the vehicle 1 or the environment of the vehicle 1, an event that has been the subject of annotation by the user will be referred to as an annotation subject event. The event here may also be referred to as a scene or a scenario.

[0077] The storage and transmission unit V4 extracts candidates to be annotation targets (hereinafter, candidate events) from events occurring in the vehicle 1 or the environment of the vehicle 1. The trigger for the extraction may include at least one of the movement of the vehicle 1 and the result of internal processing in the vehicle 1.

[0078] The trigger due to the movement of the vehicle 1 may be, for example, the occurrence of sudden deceleration, sudden acceleration, or sudden steering. The trigger due to the movement of the vehicle 1 may be a violation of a safety policy preset for the vehicle 1. The safety policy here may be, for example, a condition defined by a Responsibility Sensitive Safety (RSS) model or a Safety Force Field (SFF) model, which are models for ensuring the safety of autonomous driving. The violation of the safety policy may be, for example, a violation of a safety envelope. The violation of the safety envelope may mean that the safety envelope between the vehicle and another vehicle cannot be maintained, and the safety envelope here may mean a safety distance or an appropriate inter-vehicle distance in the RSS model. The trigger due to the movement of the vehicle 1 may be the vehicle 1 passing through a specific location or road. The specific location or road may be, for example, an intersection, a parking lot, a highway, etc. The specific location or road may be a location or road designated as an accident-prone spot in map data stored in the vehicle 1.

[0079] The trigger due to the result of the internal processing may be, for example, a poor (low confidence) mechanical annotation, which may be a classification of an object or a segmentation based on sensor data from sensors mounted on the vehicle 1.

[0080] The storage and transmission unit V4 continues to extract candidate events for a predetermined period of time. This period may be, for example, one day. The one day referred to here may be from midnight to 11:59 PM on the current day, or may be a day separated by any time. The storage and transmission unit V4 may terminate the period even if the time has not yet reached the separating point, if it determines that the vehicle 1 has finished driving for the day and will not be driven during the day. The determination that the vehicle 1 has finished driving may be made when the vehicle 1 returns home after the evening, or the behavioral patterns of the vehicle 1 may be learned by an artificial intelligence learning model and the learning model may be used to make the determination.

[0081] As the period ends, the storage and transmission unit V4 selects one or more annotation target events from the extracted candidate events. If there is one annotation target event, the burden on the user who performs the annotation may not be excessive. If there are multiple annotation target events, more vehicle data may be made useful. Furthermore, if the period is set in units of one day, the annotation may provide an opportunity for the user to reflect on the day and their driving.

[0082] The storage and transmission unit V4 may select an annotation target event based on criteria within the vehicle 1. The storage and transmission unit V4 may select an annotation target event based on criteria instructed by the server 20, or may select an annotation target event by making an inquiry to the server 20 on its own initiative. Upon receiving the inquiry, the server 20 may refer to the overall data collection status collected from multiple vehicles and notify the vehicle 1 of the annotation target event to be selected.

[0083] The selection of an event to be annotated may be performed based on whether the event is one for which a user's evaluation is required, such as a high-priority event that needs to be verified with reference to the purpose of data collection, an event for which the number or accuracy of vehicle data required for verification has not yet been obtained with reference to the data collection status up to now, etc.

[0084] The storage and transmission unit V4 generates content including questions for the user related to the selected annotation target event. This content is provided, for example, using the CID 12. The content includes question content to be displayed on the display 12a of the CID 12 and answer content that allows the user to respond to the question via the touch panel 12b.

[0085] For example, the question content is generated based on a part related to the annotation target event that cannot be accurately determined mechanically, in other words, a part for which an evaluation by the user of the vehicle 1 should be sought, in order to complement that part. Examples of questions are given below.

[0086] ・Questions after showing multiple events to be annotated: Which scene did you think was dangerous (a near miss)? ・Questions after showing an event to be annotated involving your own vehicle and an event to be annotated involving another vehicle: Compared to the example of another vehicle, which is more dangerous? ・Questions after showing multiple events to be annotated: Which scene did you find difficult or something you would not want to encounter? ・Questions after showing multiple events to be annotated: In which scene did you have to stop your car at an intersection, crosswalk, etc.? ・Questions after showing multiple events to be annotated: In which scene did an emergency vehicle pass nearby?

[0087] The answer content is generated according to the form of the question in the question content. For example, if the question is in the form of a user selecting one or more options from a plurality of options, a plurality of graphical buttons corresponding to the options are displayed on the display 12 a, and answer content is provided that allows the user to select the option by pressing the button.

[0088] The answer content may further include a user interface that allows the user to indicate whether or not to upload the vehicle data related to the annotation target event to the server 20. For example, a button to allow uploading and a button to deny uploading are displayed, and answer content that allows the user to indicate their intention by pressing the buttons is provided. The user's indication of their intention to allow uploading here may be referred to as consent to uploading.

[0089] In addition, in preparation for the user's intention to refuse uploading such answer content, it is preferable that the storage and transmission unit V4 leave the vehicle data related to the event to be annotated stored in storage 10e and suspend uploading to the server 20 until annotation is performed.

[0090] Furthermore, the storage and transmission unit V4 determines the timing for annotating the user and starts annotating the user at the determined timing, which may be, for example, when the vehicle 1 returns home and before the user gets off the vehicle 1.

[0091] After the annotation is performed, the storage transmission unit V4 selects the vehicle data to be uploaded to the server 20 based on the result of the annotation. For vehicle data for which uploading has been denied by the user, the storage transmission unit V4 prohibits uploading to the server 20 and deletes the vehicle data from the storage 10e. The storage transmission unit V4 also determines whether vehicle data for which uploading has not been denied by the user should be uploaded to the server 20. For example, the storage transmission unit V4 decides to upload to the server 20 vehicle data that has been successfully made useful through annotation and is associated with user response information. On the other hand, the storage transmission unit V4 determines vehicle data for which the making of useful data has failed as noise and decides not to upload the vehicle data to the server 20.

[0092] Next, an example of the processing method in S6 of FIG. 3 will be described in detail with reference to the flowchart of FIG.

[0093] First, in step S101, the storage and transmission unit V4 extracts candidate events for annotation from among events encountered by the vehicle 1 during a preset period. After processing step S101, in step S102, the storage and transmission unit V4 stores vehicle data related to the candidate events in the storage 10e. After processing step S102, in step S103, the storage and transmission unit V4 selects one or more annotation target events from the candidate events upon the end of the preset period.

[0094] In S104 after processing S103, the storage transmission unit V4 generates content for allowing the user of the vehicle 1 to perform annotation. In S105 after processing S104, the storage transmission unit V4 determines the timing for performing annotation. In S106 after processing S105, the storage transmission unit V4 starts the user's annotation at the timing determined in S105. Specifically, the storage transmission unit V4 provides the user with a user interface for annotating data related to the annotation target event using CID 12. To this end, the storage transmission unit V4 outputs a content provision request to CID 12.

[0095] In S107 after the process of S106, the storage and transmission unit V4 determines whether or not to upload the vehicle data based on the annotation result. If Yes, proceed to S108. If No, proceed to S109.

[0096] In S108, the storage transmission unit V4 uploads the vehicle data related to the annotated event to the server 20. The storage transmission unit V4 also refers to the free space in the storage 10e and deletes the uploaded vehicle data as necessary. The series of processes ends with S108.

[0097] In S109, the storage and transmission unit V4 prohibits uploading of vehicle data related to the annotation target event to the server 20. The storage and transmission unit V4 deletes this unuploaded vehicle data from the storage 10e. After S109, the series of processes ends.

[0098] According to the first embodiment described above, vehicle data can be made useful as a result of annotation, which is a user-initiated operation for making the data useful, realized by providing a user interface. In other words, the accuracy of the data can be improved without relying solely on mechanical processing. Therefore, it becomes possible for the server 20 to efficiently collect useful data.

[0099] Furthermore, according to the first embodiment, uploading of vehicle data related to the extracted event from the vehicle 1 to the server 20 is suspended until annotation is performed. Then, based on the result of the annotation, the vehicle data to be uploaded from the vehicle 1 to the server 20 is selected. Since vehicle data that is not suitable for data collection can be excluded, useful data can be collected efficiently.

[0100] According to the first embodiment, the annotation provides a user interface for the user to indicate their intention regarding uploading, and the user's intention is displayed. If the user indicates their intention to reject the upload in response to such an annotation, the upload is prohibited. Since vehicle data that is not suitable for data collection can be excluded, useful data can be collected efficiently.

[0101] Furthermore, according to the first embodiment, annotation candidate events or annotation target events are extracted using the movement of the vehicle 1 as a trigger. Since it is possible to focus on vehicle data related to a specific movement to be verified and make the data useful, it is possible to efficiently collect useful data.

[0102] Furthermore, according to the first embodiment, annotation candidate events or annotation target events are extracted using the result of internal processing in the vehicle 1 as a trigger. Since it is possible to focus on the internal processing to be verified and make it more useful, it becomes possible to efficiently collect useful data.

[0103] According to the first embodiment, the timing of providing the user interface is set so that the annotations provide a chance for the user to reflect on the day and their driving. This allows the vehicle data to be validated when the user's memory is likely to be fresh, and by providing the user with an opportunity to reflect on the day, the annotations can become a habit, enabling the efficient collection of a wider range of useful data.

[0104] Second Embodiment As shown in Figures 5 and 6, the second embodiment is a modification of the first embodiment. The second embodiment will be described, focusing on the differences from the first embodiment.

[0105] In the second embodiment, the vehicle 1 is communicably connected to a smartphone 40 owned by a user of the vehicle 1 through the DCM 11. As shown in Fig. 5 , the smartphone 40 is a mobile terminal including a computer, a display 40e, a touch panel 40f, a speaker, and the like, and is sized to be easily portable.

[0106] The computer in the smartphone 40 has at least one CPU 40a and one memory 40c. The memory 40c non-temporarily stores computer programs and data that can be read by the CPU 40a. The computer may also be provided with a rewritable volatile storage medium such as a RAM 40d. The CPU 40a can execute various processes in accordance with the computer programs stored in the memory 40c. The computer also includes an interface 40b for exchanging data with the outside.

[0107] The display 40e is, for example, a liquid crystal display or an organic electroluminescence display, and displays images on the display screen and, in cooperation with a speaker or the like, can present information to the user. The touch panel 40f is configured integrally with the display 40e, detects user touch operations on the display screen, and provides the detection results to the computer. In this way, the smartphone 40 also functions as an HMI device that can provide a user interface to the user.

[0108] A computer program that provides a user interface for the user to perform annotations is pre-installed in the form of an application in the memory 40c of the smartphone 40. The smartphone 40 can provide the user with a user interface for performing annotations in response to a request from the data management device 10.

[0109] In the data management device 10 of the second embodiment, the storage and transmission unit V4 selects the HMI device to be used for annotating the user, either the CID 12 or the smartphone 40. In addition to this selection, the storage and transmission unit V4 also determines the timing of annotating. The timing of annotating can be set more flexibly, taking into account the user's situation.

[0110] When the vehicle is in automated driving with an automation level of 3 or higher, the storage and transmission unit V4 may determine to perform annotation in real time using CID 12 because the burden on the driver as a user is small. That is, annotation may be performed immediately after the annotation target event occurs. In the case of real time, annotation can be performed while the user's memory is still fresh, increasing the possibility of making the vehicle data useful. On the other hand, when the vehicle is in manual driving with an automation level of 2 or lower, the burden on the driver as a user is large, so the storage and transmission unit V4 may prohibit real-time annotation and postpone it until the driver's manual driving is finished.

[0111] Furthermore, if the vehicle 1 is a vehicle capable of running using a driving battery, the storage and transmission unit V4 may determine to perform annotation using the CID 12 when the vehicle 1 is stopped to charge the driving battery. Examples of vehicles capable of running using a driving battery include battery electric vehicles (BEVs) and plug-in hybrid electric vehicles (PHEVs).

[0112] Furthermore, the storage and transmission unit V4 may determine to perform annotation using the smartphone 40 at a timing after the user gets off the vehicle 1. In this case, the storage and transmission unit V4 may determine the detailed timing, or the computer of the smartphone 40 that has received the annotation request may determine the detailed timing. The timing after the user gets off the vehicle 1 may be, for example, a timing just before the user goes to bed. The user may be able to perform annotation at a timing of their choice by notifying the user of the annotation request using a push notification function of the smartphone 40.

[0113] Next, an example of the processing method, particularly S106, among the processing shown in FIG. 4 of the first embodiment will be described in detail using the flowchart of FIG.

[0114] First, in S201, the storage and transmission unit V4 determines whether the vehicle 1 is in an autonomous driving state at automation level 3 or higher. If the determination is Yes, the process proceeds to S202. If the determination is No, the process proceeds to S203.

[0115] In S202, the storage and transmission unit V4 determines to perform annotation in real time using CID 12. After S202, the series of processes ends.

[0116] In S203, the storage and transmission unit V4 determines whether charging of the driving battery in the vehicle 1 is currently being performed or is expected to be performed in the future. The "expected" here may refer to the possibility that the driving battery will run out of power before the vehicle 1 returns home, making it impossible to drive, and therefore the user will interrupt the driving of the vehicle 1 to charge the battery. For example, during a trip, the user may want to charge the driving battery along the way (at a service area on a highway, for example). If the answer is Yes, proceed to S204. If the answer is No, proceed to S205.

[0117] In S204, the storage and transmission unit V4 uses CID 12 to determine that annotation will be performed while the driving battery is charging. The series of processes ends with S204. In S205, the storage and transmission unit V4, in cooperation with the smartphone 40, determines that annotation will be performed before the user goes to bed. The series of processes ends with S205.

[0118] According to the second embodiment described above, the timing for providing the user interface is set to when the vehicle 1 is in autonomous driving at automation level 3 or higher. When the vehicle 1 is in autonomous driving, the burden on the user related to driving is relatively small, so by performing annotation at this timing, the annoyance felt by the user can be reduced.

[0119] According to the second embodiment, the timing for providing the user interface is set to when the vehicle's driving battery is being charged. While the driving battery is being charged, the user may have to wait, so by providing annotations during such a time, the annoyance felt by the user can be reduced.

[0120] According to the second embodiment, any one of the devices 10, 20, and 30 of the vehicle data collection system DCS is communicably connected to a smartphone 40 serving as a mobile terminal used by a user. The timing for providing the user interface is set to the timing after the user gets off the vehicle 1, and a request for providing the user interface is sent to the smartphone 40. By linking with the smartphone 40, annotation can be performed at a more flexible timing without being tied down to the user's time while in the vehicle, thereby reducing the inconvenience felt by the user.

[0121] Third Embodiment As shown in the drawing, the third embodiment is a modification of the first embodiment. The third embodiment will be described, focusing on the differences from the first embodiment.

[0122] The data collection in the third embodiment is used to improve applications used in the vehicle 1. Here, the applications may be, for example, an image processing application used in the image processing device 14, a behavior prediction application used in the behavior prediction device 15, an occupant state analysis application used in the occupant state sensor 16, etc.

[0123] 7, the image processing device 14 is an in-vehicle device that processes images captured by the external environment camera 13. The external environment camera 13 is one of the autonomous sensors mounted on the vehicle 1. The external environment camera 13 is attached to an upper portion of the front windshield inside the vehicle cabin, for example, and is positioned so as to be able to capture images of the forward external environment of the vehicle 1. The external environment camera 13 may be capable of capturing color images or monochrome images.

[0124] The image processing device 14 may be installed in the same housing as the external environment camera 13 and may constitute a camera module integrated with the external environment camera 13. The image processing device 14 may be configured separately from the external environment camera 13 and may be communicatively connected to the external environment camera 13 via wireless communication such as an in-vehicle LAN or wired communication such as a wire harness.

[0125] The image processing device 14 may be, for example, an ECU mainly composed of a computer. The computer constituting the image processing device 14 has at least one CPU 14a and one memory 14c. The memory 14c non-temporarily stores computer programs and data that can be read by the CPU 14a. The computer may also be provided with a rewritable volatile storage medium such as a RAM 14d. The CPU 14a can execute various processes in accordance with the computer programs stored in the memory 14c. The computer also includes an interface 14b for exchanging data with the outside.

[0126] An image processing program as a computer program is installed in the image processing device 14 in the form of an image processing application. The CPU 14a may load the image processing program to perform object detection, which detects objects reflected in an image. The CPU 14a may load the image processing program to perform semantic segmentation, which identifies and labels pixels as classes.

[0127] The behavior prediction device 15 is an in-vehicle device that predicts the behavior of other road users in the external environment of the vehicle 1 (for example, around the vehicle 1) based on the image processing results by the image processing device 14 and other information.

[0128] The behavior prediction device 15 may be, for example, an ECU mainly composed of a computer. The computer constituting the behavior prediction device 15 has at least one CPU 15a and one memory 15c. The memory 15c non-temporarily stores computer programs and data that can be read by the CPU 15a. The computer may also be provided with a rewritable volatile storage medium such as a RAM 15d. The CPU 15a can execute various processes in accordance with the computer programs stored in the memory 15c. The computer also includes an interface 15b for exchanging data with the outside.

[0129] A behavior prediction program as a computer program is installed in the behavior prediction device 15 in the form of a behavior prediction application. The CPU 15a may predict the behavior of other road users, such as other vehicles, around the vehicle 1 by loading the behavior prediction program. The behavior of the other vehicles may be the speed, acceleration, trajectory, etc. predicted for the other vehicles (after a predetermined time). The speed may be a speed range including one or both of a predicted minimum speed limit and a predicted maximum speed limit. The acceleration may be an acceleration range including one or both of a predicted minimum acceleration limit and a predicted maximum acceleration limit. The trajectory may be a trajectory of traveling straight in the current lane, a trajectory of changing lanes, a trajectory of turning right or left, a trajectory of avoiding an obstacle, a trajectory of stopping on the shoulder of the road, etc. The CPU 15a may also predict the behavior of motorcycles, bicycles, pedestrians, animals, etc. around the vehicle 1.

[0130] The occupant status sensor 16 is a sensor that detects the status of an occupant (e.g., a driver) of the vehicle 1, and is sometimes referred to as a driver status monitor (DSM). The occupant status sensor 16 includes an in-vehicle camera 16e and a computer. The in-vehicle camera 16e is, for example, an infrared camera, and is configured to be able to capture an image of an area in the vehicle interior that includes the face of the occupant.

[0131] The computer in the occupant status sensor 16 has at least one CPU 16a and one memory 16c. The memory 16c non-temporarily stores computer programs and data that can be read by the CPU 16a. The computer may also be provided with a rewritable volatile storage medium such as a RAM 16d. The CPU 16a can execute various processes in accordance with the computer programs stored in the memory 16c. The computer also includes an interface 16b for exchanging data with external devices.

[0132] An occupant state analysis program as a computer program is installed in the form of an occupant state analysis application in the occupant state sensor 16. By reading the occupant state analysis program, the CPU 16a may analyze the driver's emotions from the driver's facial expressions as the occupant state.

[0133] In the third embodiment, the storage and transmission unit V4 of the data management device 10 sequentially uploads vehicle data related to processing in each device 14, 15, and 16 to the server 20, assuming that comprehensive consent regarding uploading vehicle data to the server 20 has been obtained in advance. Here, sequential uploading may mean that the vehicle data is uploaded immediately after it is acquired or generated by the data management device 10, except in cases where uploading is technically impossible due to a poor communication environment. Here, the vehicle data may be uploaded to the server 20 with information about the scene (scene information) attached. Here, the scene information is information that represents the scene in which the vehicle data was generated and is used to select an event to be annotated. As an example, the scene information is information indicating that the data was generated when the vehicle 1 passed through an intersection.

[0134] On the other hand, the data collection unit C3 of the server 20 selects annotation target events based on the collection status of vehicle data to which scene information has been added during a preset period. This selection may be performed in a two-stage process, as in the first embodiment, in which candidate events are extracted and then the candidate events are narrowed down to the annotation target events. Alternatively, the selection may be performed in a one-stage process from all the scene information.

[0135] The data collection unit C3 then generates content including questions for the user related to the selected annotation target events. The data collection unit C3 may generate the content by referencing the acquisition instruction information for the multiple vehicles generated by the instruction setting unit C2. The data collection unit C3 may also generate content by acquiring the verification results from the verification unit D4 of the development computer 30, so as to complement aspects of the application that were difficult or insufficient to verify.

[0136] The content includes question content and answer content, as in the first embodiment. However, in data collection aimed at improving an application, it is preferable that the data collection unit C3 selects events that could not be handled in the processing of the application or events that were processed with low accuracy as events to be annotated, and generates questions whose answers from the user lead to the correct answer for the processing.

[0137] Specific examples of content are shown in Figures 8 to 11. The example in Figure 8 aims to improve an algorithm used in an image processing application. The data collection unit C3 acquires images captured by the external environment camera 13 and detection results of objects reflected in the images as vehicle data related to a scene in which object detection was unsuccessful through image processing by the image processing device 14. The data collection unit C3 then generates content to allow the user to perform annotation on the scene.

[0138] Specifically, the data collection unit C3 generates a tile image UI1a by dividing the image of this scene into tiles and arranging them in a grid pattern. Each tile in the tile image UI1a can be switched between a selected state and a non-selected state by the user's touch operation. The data collection unit C3 generates a request text UI1b that expresses a request to the user regarding the tile image UI1a. For example, this request text UI1b may be, "Please select all tiles that show cars. When you have finished selecting all, press the OK button." The data collection unit C3 generates a button UI1c that the user can use to indicate that they have completed their selection.

[0139] The example of FIG. 9 also aims to improve an algorithm used in an image processing application. Specifically, the data collection unit C3 generates a marking image UI2a by adding markings (e.g., frames) to objects whose types could not be identified in an image captured by the external environment camera 13. The data collection unit C3 generates a request text UI2b that expresses a request to the user regarding the marking image UI2a. For example, this request text UI2b may be a sentence such as, "Please select from the options the type of object shown in the marking in the scene in the image below." The data collection unit C3 further generates an answer interface UI2c that displays options for the object type. The options may be, for example, "bicycle," "motorcycle," "tricycle," "stroller," "handcart," or "none of the above."

[0140] 10 is intended to improve the algorithm used in the behavior prediction application. The data collection unit C3 acquires images captured by the external environment camera 13 and the behavior prediction results of other road users as vehicle data related to a scene in which the behavior prediction device 15 failed to predict the other road users. The data collection unit C3 then generates content to allow the user to annotate this scene.

[0141] Specifically, the data collection unit C3 generates a marking image UI3a by adding a marking (e.g., a frame) to the image captured by the external environment camera 13 for other road users whose behavior could not be predicted. The data collection unit C3 generates a request sentence UI3b that expresses a request to the user regarding the marking image UI3a. The request sentence UI3b may be, for example, "What was the pedestrian in the marking in the scene in the image below trying to do?" The data collection unit C3 further generates an answer interface UI3c that shows the road user options for predicted behavior. The options may be, for example, "Trying to cross in front of the vehicle," "Walking toward the crosswalk behind," "Looking for something that had fallen on the road," or "None of these apply."

[0142] 11 is intended to improve the algorithm used in the occupant state analysis application. The data collection unit C3 acquires vehicle data related to a scene in which the occupant state sensor 16 failed to analyze the driver's emotion, including an image of the driver's facial expression captured by the interior camera 16e, the analysis result of the driver's emotion, and the position of the vehicle 1 when the image was captured. The data collection unit C3 then generates content to allow the user to annotate the scene.

[0143] Specifically, the data collection unit C3 generates a map image UI4a showing the position of the vehicle 1 in the scene. The data collection unit C3 generates a response interface UI4c showing options for candidate driver emotions in the scene. The data collection unit C3 generates a request sentence UI4b that expresses a request to the user in writing regarding the map image UI4a and the response interface UI4c. The request sentence UI4b may be, for example, "How did you feel when you went straight through the intersection you just passed (see map)?" The options may be "I was annoyed by the vehicle ahead," "I was curious about the buildings along the road," "I was sleepy," or "none of these apply."

[0144] The data collection unit C3 in the server 20 transmits the generated content to the vehicle 1 and requests the vehicle 1 to perform annotation by the user. The storage transmission unit V4 in the vehicle 1 uses the CID 12 to provide a user interface for the user to perform annotation based on the request. The storage transmission unit V4 adds answer information to the vehicle data in accordance with the user's answer. The storage transmission unit V4 uploads the vehicle data with the answer information added to the server 20, and the data collection unit C3 updates the vehicle data.

[0145] The vehicle data to which the answer information has been added in this way may be used for machine learning by the machine learning unit D3, thereby improving the application.

[0146] The content generated by the data collection unit C3 may not be content that provides the user with options, but may be content that allows the user to respond by freely inputting language. For example, the content may be content that allows the user to respond in the form of a sentence in a language input by keyboard operation, or content that allows the user to respond by voice input in a language.

[0147] When a user answers using free language input, the answer information may be input to an artificial intelligence language model and converted into a format that is easy to process in at least one of a scene extraction unit D1, a labeling unit D2, and a machine learning unit D3. The language model may be CHATGPT (registered trademark), or a model using the same that is optimized for the conversion.

[0148] Conversion into a form that facilitates data processing may mean summarizing the linguistic information input by the user into abbreviated linguistic information, or may mean converting the linguistic information input by the user into numerical values ​​or the like used for data processing. The conversion using a language model may be performed by the storage and transmission unit V4 in the vehicle 1, by the data collection unit C3 in the server 20, or by any one of the scene extraction unit D1, the labeling unit D2, and the machine learning unit D3 in the development computer.

[0149] Next, an example of the processing method in S6 of Fig. 3 in the third embodiment will be described in detail using the flowchart of Fig. 12. First, in S301, the storage and transmission unit V4 of the vehicle 1 sequentially uploads vehicle data to the server 20.

[0150] In S302 after processing S301, the data collection unit C3 in the server 20 sequentially stores the vehicle data collected from the vehicle 1 in the storage 20e. In S303 after processing S302, the data collection unit C3 extracts candidate events to be annotation targets based on the collected vehicle data. In S304 after processing S303, the data collection unit C3 selects annotation target events from the candidate events. In S305 after processing S304, the data collection unit C3 generates content related to the selected annotation target events. In S306 after processing S305, the data collection unit C3 requests the vehicle 1 to perform annotation.

[0151] In S307 after the processing of S306, the storage transmission unit V4 in the vehicle 1 determines the timing of performing annotation. In S308 after the processing of S307, the storage transmission unit V4 performs annotation using CID 12. In S309 after the processing of S308, the storage transmission unit V4 uploads the annotation results to the server 20. The series of processes ends with S309.

[0152] According to the third embodiment described above, vehicle data is sequentially uploaded from the vehicle 1 to the server 20. The uploaded vehicle data is then stored in the storage 20e, which serves as a storage medium, in the server 20. The vehicle data stored in the storage 20e is updated based on the results of annotation performed after the upload. Because the vehicle data is once collected by the server 20, loss of vehicle data due to individual judgment by the vehicle 1 is prevented. Since the vehicle data can be updated to be useful based on an overall judgment after collection by the server 20, it is possible to efficiently collect useful data.

[0153] According to the third embodiment described above, in the annotation, a user interface is provided for requesting a user's judgment on an event that is a weakness of an application used in the vehicle 1, and the user responds. Then, the vehicle data related to the annotation target event is uploaded from the vehicle 1 to the server 20 with the user's response information associated with the vehicle data. Since it becomes possible to verify application weaknesses using events actually encountered by the vehicle 1 and the user's responses to those events, the usefulness of the vehicle data is dramatically increased.

[0154] According to the third embodiment described above, machine learning is performed using vehicle data associated with response information, and parameters used in an application are optimized. Because the response information provided by the user is used for machine learning, the usefulness of the vehicle data is further increased.

[0155] 13, the fourth embodiment is a modification of the third embodiment. The fourth embodiment will be described, focusing on the differences from the third embodiment.

[0156] In the fourth embodiment, the data management device 410 may further include an advice generating unit V5 as a functional unit realized by the CPU 10a executing a computer program stored in the memory 10c.

[0157] When the storage and transmission unit V4 decides to perform annotation, the advice generation unit V5 generates advice content to be added to the content for annotation. For example, the advice content may be a piece of advice about a scene that is the annotation target event. For example, if the scene occurred during manual driving by the driver, the piece of advice may include advice about driving by the user.

[0158] The driving advice provided by the user may be advice for improving the safety of manual driving. The driving advice provided by the driver may be advice for the user to comply with laws and regulations such as the Road Traffic Act. The driving advice provided by the user may be advice to indicate differences between the user of the vehicle 1 and an exemplary driver.

[0159] The advice may be an introduction (explanation) of a function of a driving assistance application implemented in the vehicle 1. The advice may be an introduction (explanation) of how to use the driving assistance application. The driving assistance application may be, for example, Adaptive Cruise Control (ACC), an application for assisting in maintaining a vehicle distance, or Lane Keeping Assist (LKA), an application for assisting in keeping a lane.

[0160] Furthermore, when the vehicle data collection system DCS tests a development application in a special mode such as a shadow mode or a ghost mode to collect vehicle data, data collection may be difficult due to a user. For example, data collection may be difficult if the user's driving is abnormal and deviates from the data collection conditions in the test. In this case, the advice may be to request the user to drive in a manner consistent with the data collection conditions.

[0161] According to the fourth embodiment described above, the user interface includes content for providing advice to the user in addition to content for allowing the user to perform annotation. By providing advice to the user when performing annotation, it is possible to reduce the inconvenience felt by the user and encourage the user to change their behavior.

[0162] (Other Embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0163] In other embodiments, the period used as a division for extracting candidate events may be one hour, half a day, multiple days, one week, one month, or the like, other than one day.

[0164] In another embodiment related to the third embodiment, the data collection unit C3 of the server 20 may not generate content related to the annotation target event. The data collection unit C3 may instruct the vehicle 1 to generate content for the annotation target event selected by itself, and the storage and transmission unit V4 of the vehicle 1 may generate the content based on the instruction.

[0165] As another embodiment related to the third embodiment, the image processing device 14 may process images obtained by an external environment sensor other than the external environment camera 13. For example, the image processing device 14 may process images generated by scanning with LiDAR (light detection and ranging / laser imaging detection and ranging).

[0166] 8 and 9 of the third embodiment, the storage and transmission unit V4 may not upload the image captured by the external environment camera 13 as is. The storage and transmission unit V4 may compress the image, reduce the resolution, or perform other processing to reduce the data size before uploading it.

[0167] As another embodiment related to Figures 8 to 10 of the third embodiment, the object or road user to which the user is prompted for an answer in the annotation may be a random object or road user, rather than an object or road user for which processing has not been successful. Furthermore, the events to be annotated may also be selected randomly. Events that are recognized as being processing-safe on the vehicle 1 side are randomly extracted, and by having the user check their answers, it is possible to discover weaknesses in the application that have not been discovered in the past.

[0168] In another embodiment, user annotation may be performed with the intention of having the user confirm the results when multiple processes produce inconsistent results. The inconsistency in the results of multiple processes corresponds to a trigger due to the results of internal processing. For example, when the detection results of multiple external environment sensors (e.g., a camera and LiDAR, a camera and millimeter-wave radar, etc.) do not match, the content may prompt the user to answer which external environment sensor's detection result is correct. Furthermore, for example, when the processing results of multiple on-board devices that configure redundant systems and perform similar processes do not match, the content may prompt the user to confirm the correct processing result in order to identify the on-board device experiencing an abnormality.

[0169] In another embodiment, the user's annotation may be performed with the intention of having the user check the state of the vehicle interior. For example, the content may prompt the user to answer questions about the classification of passengers. The classification of passengers may be the number of passengers, the age of the passengers, the gender of the passengers, whether the passengers are human, pet, or the like. Also, for example, the content may prompt the user to answer questions about luggage in the vehicle 1. The content may prompt the user to answer questions about the weight of the luggage, whether the luggage is susceptible to shaking and vibration, and the like.

[0170] In another embodiment, the vehicle 1 may be a right-hand drive vehicle or a left-hand drive vehicle. Furthermore, the traffic environment in which the vehicle 1 travels may be a traffic environment where traffic is assumed to be on the left side of the road, or a traffic environment where traffic is assumed to be on the right side of the road. The data management device 10 and the server 20 according to the present disclosure may be optimized as appropriate, taking into account the road traffic laws, customs, data (protection) laws, etc. of each country and region.

[0171] In another embodiment, at least one of the computers constituting the data management device 10, the image processing device 14, the behavior prediction device 15, the occupant status sensor 16, and the smartphone 40 may be equipped with another type of processor (e.g., a GPU) instead of or in addition to a CPU.

[0172] In another embodiment, at least one of the computers constituting the data management device 10, the image processing device 14, the behavior prediction device 15, the occupant status sensor 16, and the smartphone 40 may be a SoC (System on a Chip) that integrates a processor such as a CPU, a memory, an interface, and the like on a single chip, or may have at least one SoC as a component of the computer. Furthermore, a circuit such as an FPGA may be incorporated into part of the SoC, and in this case, the processor may control the circuit.

[0173] Furthermore, at least one of the data management device 10, the server 20, and the development computer 30 may be configured to execute processing by a circuit, rather than by a processor such as a CPU.

[0174] In this disclosure or claims, "at least one of the circuit and the processor performs a function that causes a user interface to be provided" includes the case where only the circuit performs the function that causes a user interface to be provided. Also, "at least one of the circuit and the processor performs a function that causes a user interface to be provided" includes the case where only the processor performs the function that causes a user interface to be provided. Furthermore, "at least one of the circuit and the processor performs a function that causes a user interface to be provided" includes the case where the circuit performs some of the functions that cause a user interface to be provided and the processor performs the remaining functions.

[0175] (Disclosure of Technical Ideas) This specification discloses multiple technical ideas described in the following multiple clauses. Some clauses may be described in a multiple dependent form, where the subsequent clause alternatively cites the preceding clause. These multiple dependent clauses define multiple technical ideas.

[0176] <Technical Idea 1> A vehicle data collection system including at least one processing unit (10a, 20a, 30a) that collects data of a vehicle (1) in a server (20), wherein the at least one processing unit is configured to: extract events that occur in the vehicle or in the environment in which the vehicle is located, and that allow a user of the vehicle to perform a useful operation to increase the usefulness of the data; and cause at least one device (12, 40) to provide a user interface for allowing a user of the vehicle to perform the useful operation on the data related to the extracted events.

[0177] <Technical Idea 2> The vehicle data collection system according to Technical Idea 1, wherein the at least one processing unit is further configured to: suspend uploading of the extracted event-related data from the vehicle to the server until the utilization operation is executed; and select the data to be uploaded from the vehicle to the server based on a result of the utilization operation.

[0178] <Technical Idea 3> The utilization operation includes providing the user interface for the user to express their intention regarding the upload and allowing the user to express their intention, and the at least one processing unit prohibits the upload when the user expresses their intention to reject the upload during the selection process. This is the vehicle data collection system described in Technical Idea 2.

[0179] <Technical Idea 4> The vehicle data collection system according to Technical Idea 1, wherein the at least one processing unit is further configured to: sequentially upload the data from the vehicle to the server; store the uploaded data in a storage medium (20c) at the server; and update the data stored in the storage medium based on a result of the utilization operation performed after the upload.

[0180] <Technical Concept 5> The vehicle data collection system according to any one of Technical Concepts 1 to 4, wherein the at least one processing unit extracts the event using a movement of the vehicle as a trigger in the extracting.

[0181] <Technical Concept 6> The vehicle data collection system according to any one of Technical Concepts 1 to 5, wherein the at least one processing unit extracts the event in response to a result of internal processing in the vehicle as a trigger.

[0182] <Technical Idea 7> The vehicle data collection system according to any one of Technical Ideas 1 to 6, wherein the usefulness operation includes, for an event for which the user's evaluation should be requested, providing the user interface for requesting the user's evaluation and having the user respond, and the at least one processing unit is further configured to associate the user's response information with the data related to the event and upload the data from the vehicle to the server.

[0183] <Technical Idea 8> The vehicle data collection system described in any one of Technical Ideas 1 to 7, wherein the usability operation includes providing the user interface for requesting a judgment result from the user in an event that is a weak point of an application used in the vehicle, and having the user respond, and the at least one processing unit is further configured to associate the user's response information with the data related to the event, and upload the data from the vehicle to the server.

[0184] <Technical Idea 9> The vehicle data collection system according to Technical Idea 8, wherein the at least one processing unit is further configured to perform machine learning using the data associated with the response information, and optimize parameters used in the application.

[0185] <Technical Idea 10> The vehicle data collection system according to any one of Technical Ideas 1 to 9, wherein the at least one processing unit is further configured to set a timing for providing the user interface so that the utilization operation serves as a daily review for the user.

[0186] <Technical Idea 11> The vehicle data collection system according to any one of Technical Ideas 1 to 9, wherein the at least one processing unit is further configured to set the timing for providing the user interface to a timing when the vehicle is in autonomous driving at automation level 3 or higher.

[0187] <Technical Idea 12> The vehicle data collection system according to any one of Technical Ideas 1 to 9, wherein the at least one processing unit is further configured to set the timing for providing the user interface to a timing when a driving battery of the vehicle is being charged.

[0188] <Technical Idea 13> A vehicle data collection system described in any one of Technical Ideas 1 to 10, which is communicatively connected to a mobile terminal (40) used by the user, and wherein the at least one processing unit sets the timing for providing the user interface to a timing after the user gets off the vehicle, and requests the mobile terminal as the at least one device to provide the user interface.

[0189] <Technical Idea 14> The vehicle data collection system according to any one of Technical Ideas 1 to 11, wherein the user interface includes content for providing advice to the user in addition to content for causing the user to perform the useful operation.

Claims

1. A vehicle data collection system comprising at least one processing unit (10a, 20a, 30a) that collects data of a vehicle (1) in a server (20), wherein the at least one processing unit is configured to: extract events that occur in the vehicle or in the environment in which the vehicle is located, and allow a user of the vehicle to perform a usability improvement operation to increase the usability of the data; and cause at least one device (12, 40) to provide a user interface for performing the usability improvement operation on the data related to the extracted events.

2. The vehicle data collection system of claim 1, wherein the at least one processing unit is further configured to: suspend uploading of the extracted data related to the event from the vehicle to the server until the utilization operation is executed; and select or reject the data to be uploaded from the vehicle to the server based on the result of the utilization operation.

3. The vehicle data collection system of claim 2, wherein the utilization operation includes providing a user interface for the user to express their intention regarding the upload and allowing the user to express their intention, and wherein the at least one processing unit prohibits the upload if, in the selection process, the user expresses their intention to refuse the upload.

4. The vehicle data collection system of claim 1, wherein the at least one processing unit is further configured to: sequentially upload the data from the vehicle to the server; store the uploaded data in a storage medium (20c) at the server; and update the data stored in the storage medium based on the result of the utilization operation performed after the upload.

5. The vehicle data collection system according to claim 1, wherein the at least one processing unit extracts the event using the movement of the vehicle as a trigger in the extracting step.

6. The vehicle data collection system according to claim 1 or 5, wherein the at least one processing unit extracts the event using the result of internal processing in the vehicle as a trigger in the extracting step.

7. The vehicle data collection system of claim 1, wherein the utility operation includes providing the user interface for requesting the user's evaluation in an event for which the user's evaluation should be requested and having the user respond, and the at least one processing unit is further configured to associate the user's response information with the data related to the event and upload the data from the vehicle to the server.

8. The vehicle data collection system of claim 1, wherein the usability operation includes providing the user interface for requesting a judgment result from the user in an event that is a weakness of an application used in the vehicle and having the user respond, and the at least one processing unit is further configured to associate the user's response information with the data related to the event and upload the data from the vehicle to the server.

9. The vehicle data collection system of claim 8, wherein the at least one processing unit is further configured to: perform machine learning using the data associated with the response information to optimize parameters used in the application.

10. The vehicle data collection system of claim 1, wherein the at least one processing unit is further configured to: set a timing for providing the user interface so that the usefulness operation is a daily review for the user.

11. The vehicle data collection system of claim 1, wherein the at least one processing unit is further configured to set the timing for providing the user interface to a timing when the vehicle is undergoing automated driving at an automation level of 3 or higher.

12. The vehicle data collection system according to claim 1, wherein the at least one processing unit is further configured to set the timing for providing the user interface to a timing when the vehicle is charging a traction battery.

13. The vehicle data collection system according to claim 1, which is communicatively connected to a mobile terminal (40) used by the user, and wherein the at least one processing unit sets the timing for providing the user interface to a timing after the user gets off the vehicle, and requests the mobile terminal as the at least one device to provide the user interface.

14. The vehicle data collection system according to claim 1, wherein the user interface includes content that provides advice to the user in addition to content that causes the user to perform the usefulness operation.

15. A vehicle data management device equipped with at least one processing unit (10a), mounted on a vehicle (1) communicatively connected to a server (20), and configured to upload vehicle data collected by the server from the vehicle to the server, wherein the at least one processing unit is configured to: extract events that occur in the vehicle or the environment in which the vehicle is located, and that allow the user of the vehicle to perform usefulness operations to increase the usefulness of the data; and cause at least one device (12, 40) to provide a user interface for performing the usefulness operations on the data related to the extracted events.

16. A vehicle data collection device comprising at least one processing unit (20a), communicatively connected to a vehicle (1), and configured to collect data of the vehicle, wherein the at least one processing unit is configured to: acquire the data and extract, from among events occurring in the vehicle or the environment in which the vehicle is located, events that allow a user of the vehicle to perform a usability-enhancing operation to increase the usability of the data; and cause at least one device (12, 40) to provide a user interface for allowing a user to perform the usability-enhancing operation on the data related to the extracted events.

Images