Systems and methods for continuous verification of verification-as-a-service for advanced driver assistance systems and autonomous vehicles

Abstract

The present invention relates to systems and methods for verification-as-a-service that enables continuous verification for advanced driver assistance systems and autonomous vehicles. A motor vehicle continuous verification system includes a backend that collects data from a vehicle fleet and wirelessly communicates with the vehicle fleet. The backend wirelessly communicates with at least one customer. A vehicle module is provided on-board each motor vehicle of a plurality of motor vehicles of the vehicle fleet and performs on-board vehicle verification analysis. A fleet-based verification module that manages data defining configurations and capabilities of the plurality of motor vehicles of the vehicle fleet is provided at the backend or cloud-based. A verification manager generates verification tasks based on user-defined or desired outcomes of verification analysis and vehicle fleet availability. A client module, which is remote with respect to the plurality of motor vehicles of the vehicle fleet, has interface items applied by at least one customer seeking to perform verification analysis.

Description

Technical Field

[0001] This disclosure relates to the verification of motor vehicle operation and monitoring systems. Background Technology

[0002] Motor vehicle operation and monitoring systems require validation to confirm proper functioning. Traditionally, this involves constructing new vehicle models or types and running them for extended periods to capture operational data from multiple different operating scenarios, including city driving, highway driving, and operation under varying weather conditions. This approach is extremely time-consuming and requires collecting an exceptionally large amount of data to validate the vehicle or category. Camera-based forward collision warning systems can be validated by prolonged driving on roads. While the testing is extensive, it is impossible to cover all driving scenarios during this type of road-driving validation method.

[0003] Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AV) features need to operate in highly complex environments with uncertainties and during rare events that can be extremely challenging using the conventional validation methods described above. Thorough testing and validation of these features therefore necessitates continuous operation of these features, requiring extensive data collection and annotation as mentioned above.

[0004] Therefore, although current rear-view camera systems for motor vehicles have been proven to achieve their intended purpose, there remains a need for new and improved camera-based adaptive taillight systems. Summary of the Invention

[0005] According to several aspects, a continuous vehicle verification system includes a backend that collects data from and wirelessly communicates with a vehicle fleet. The backend also wirelessly communicates with at least one client. A vehicle module is provided for performing onboard vehicle verification analysis on each of the multiple vehicles in the vehicle fleet. Alternatively, a fleet-based verification module, either at the backend or in the cloud, manages data defining the configuration and capabilities of the multiple vehicles in the vehicle fleet. A verification manager generates verification tasks based on user-defined or expected outputs of verification tasks and fleet vehicle availability. Remote client modules, relative to the multiple vehicles in the vehicle fleet, have an interface for at least one client application seeking to perform verification analysis.

[0006] In another aspect of this disclosure, the vehicle module includes: a wireless communication component that manages communication between a backend and various vehicle systems of multiple motor vehicles; and a task manager that receives tasks, queries, and commands, schedules the execution of tasks, queries, and commands based on predefined priorities, interacts with vehicle users, and forwards the results of verified task requests, requested and generated data, and vehicle configuration files to the backend.

[0007] In another aspect of this disclosure, a human-machine interface (HMI) is provided, in which a task manager also communicates with the HMI and is responsible for processing override signals received via the HMI from one or more users of a vehicle fleet.

[0008] In another aspect of this disclosure, the task manager communicates with the wireless communication module.

[0009] In another aspect of this disclosure, the wireless communication component receives verification tasks, commands, and queries from the backend and forwards the verification analysis results and a set of vehicle profiles from the vehicle fleet to the backend.

[0010] In another aspect of this disclosure, the wireless client communication uplink uploads data and requests for vehicle fleet verification data (including verification requests, task definitions, and job queries) to the backend via the wireless client communication uplink. The wireless client communication downlink, containing data including vehicle operating status, verification job progress, verification results, and costs, returns data to the backend via the wireless client communication downlink.

[0011] In another aspect of this disclosure, the client module includes a client interface that monitors and controls client requests to the fleet-based verification module and returns verification data to at least one client.

[0012] In another aspect of this disclosure, the fleet vehicle manager communicates with the fleet vehicle database and communication devices. The fleet vehicle manager maintains fleet vehicle information, including adding new vehicles to the fleet vehicle database, removing old or discontinued vehicles, managing vehicle capability information, and updating new vehicle features and capabilities.

[0013] In another aspect of this disclosure, the vehicle fleet comprises multiple motor vehicles of various brands and models operating in any location in the world; and at least one customer includes service users, suppliers, original equipment manufacturers (OEMs), and government agencies.

[0014] In another aspect of this disclosure, the simulator communicates with the verification manager, thereby allowing the task to be profiled before it is deployed to confirm whether a vehicle that may request the task has the capability to perform the required verification analysis.

[0015] According to several aspects, a continuous verification system for motor vehicles includes a backend that collects data from and wirelessly communicates with at least one motor vehicle, and also wirelessly communicates with at least one client. A vehicle module is provided for mounting on at least one motor vehicle and performing verification analysis. A verification manager generates verification tasks for the verification analysis based on user definitions or the expected output of the verification task and the availability of at least one motor vehicle. A simulator performs a profiling analysis on the verification task before it is deployed to confirm whether the at least one motor vehicle requesting the verification task has the capability to perform the verification analysis. A client module is remote relative to at least one motor vehicle and has an interface project applied by at least one client seeking to perform the verification analysis.

[0016] In another aspect of this disclosure, at least one motor vehicle defines a fleet of multiple motor vehicles.

[0017] In another aspect of this disclosure, the fleet-based verification module communicates with the verification manager and, either at the backend or in the cloud, manages data defining the configuration and capabilities of multiple vehicles in a vehicle fleet.

[0018] In another aspect of this disclosure, the wireless communication module receives verification tasks, queries, and commands, schedules the execution of verification tasks based on a predefined priority plan, and interacts with the vehicle user. In response to a verification request, the wireless client communication downlink returns data including vehicle operating status, verification operation progress, verification results, and costs from the backend via the wireless client communication downlink.

[0019] In another aspect of this disclosure, the task manager communicates with the wireless communication module. The job manager communicates with the task manager. The job executor is responsible for performing the job and controlling multiple sensor gateways and vehicle systems, such that data is received by the job executor.

[0020] In another aspect of this disclosure, the configuration file manager communicates with the task manager and job executors and manages vehicle capability configuration files, which include sensor configurations, available sensor types, available vehicle types, and maximum available resources. The configuration file manager also manages runtime configuration files that include resource usage and task and job statistics.

[0021] In another aspect of this disclosure, a first gateway defining a sensor gateway communicates with a job actuator and a job manager. The sensor gateway also communicates with multiple onboard sensors of at least one motor vehicle, which provide sensing signals for the operating environment of the at least one motor vehicle. A second gateway defining a vehicle system gateway communicates with the job manager.

[0022] According to several aspects, a method for performing continuous verification of a vehicle fleet includes: collecting data from and wirelessly communicating with the vehicle fleet via a backend, the backend also wirelessly communicating with at least one client; performing onboard vehicle verification analysis using vehicle modules provided for each of the multiple vehicles in the vehicle fleet; managing data defining the configuration and capabilities of the multiple vehicles in the vehicle fleet using or provided at the backend or in the cloud-based verification module; generating verification tasks using a verification manager based on the expected output of the verification tasks defined by the user or the verification analysis and the availability of vehicles in the fleet; and having at least one client seeking to perform the verification analysis apply the application interface project via a client module, wherein the client module is remote relative to the multiple vehicles in the vehicle fleet.

[0023] In another aspect of this disclosure, the method also includes maintaining fleet vehicle information using a fleet vehicle manager that communicates with a fleet vehicle database and communication equipment, including adding new vehicles to the fleet vehicle database, removing old or discontinued vehicle data from the vehicle fleet, managing vehicle capability information, and updating new vehicle features and capabilities.

[0024] In another aspect of this disclosure, the method also includes using a simulator that communicates with the verification manager to perform a profiling analysis on the verification task prior to its deployment to confirm whether at least one of a plurality of motor vehicles that may request the verification task has the capability to perform the verification analysis.

[0025] This invention includes the following technical solutions:

[0026] Option 1. A continuous verification system for motor vehicles, comprising:

[0027] The backend collects data from and communicates wirelessly with the vehicle fleet, and also communicates wirelessly with at least one customer.

[0028] A vehicle module, which is provided to be mounted on each of the multiple motor vehicles in the vehicle fleet and to perform on-board vehicle verification analysis;

[0029] A fleet-based verification module, which is provided either at the backend or in the cloud, manages data defining the configuration and capability profiles of multiple vehicles in the vehicle fleet.

[0030] A validation manager generates multiple validation tasks based on user-defined or expected outputs of multiple validation tasks included in a validation analysis and fleet vehicle availability; and

[0031] A client module, which is remote relative to multiple vehicles in the vehicle fleet, has an interface item provided by at least one client application seeking to perform the verification analysis.

[0032] Option 2. The continuous verification system for motor vehicles according to Option 1, wherein the vehicle module includes:

[0033] A wireless communication component that manages communication between the backend and the various vehicle systems of the plurality of motor vehicles; and

[0034] The task manager receives the multiple verification tasks, multiple queries, and multiple commands, schedules the execution of the multiple verification tasks, multiple queries, and multiple commands based on a predefined priority plan, interacts with the vehicle user, and forwards the results of the verification task requests, the requested and generated data, and a set of vehicle configuration files to the backend.

[0035] Option 3. The continuous verification system for motor vehicles according to Option 2 further includes a human-machine interface (HMI), wherein the task manager also communicates with the HMI and is responsible for processing override signals received via the HMI from one or more users of the motor vehicles in the vehicle fleet.

[0036] Option 4. The continuous verification system for motor vehicles according to Option 2, wherein the task manager communicates with the wireless communication component; and further includes a verification request management component that authorizes the at least one client to generate an estimate of the required computing resources necessary to respond to a client request for data.

[0037] Option 5. The continuous verification system for motor vehicles according to Option 2, wherein the wireless communication component receives the plurality of verification tasks, the plurality of queries and the plurality of commands from the backend, and forwards the verification analysis results and the set of vehicle configuration files from the vehicle fleet to the backend.

[0038] Option 6. The continuous verification system for motor vehicles according to Option 1, comprising:

[0039] A wireless client communication uplink, wherein the at least one client communicates with the wireless client communication uplink to wirelessly upload data and requests for vehicle fleet verification data to the backend via the wireless client communication uplink; the uploaded data and requests include multiple verification requests, task definitions, and job queries; and

[0040] The wireless client communication downlink includes data on vehicle operating status, verification operation progress, verification results, and costs, which are returned to the backend via the wireless client communication downlink.

[0041] Option 7. The continuous verification system for motor vehicles according to Option 1, wherein the client module includes a client interface for monitoring and controlling requests sent from the at least one client to the fleet-based verification module and verification data returned to the at least one client.

[0042] Option 8. The continuous verification system for motor vehicles according to Option 1 further includes a fleet vehicle manager, which communicates with a fleet vehicle database and communication equipment. The fleet vehicle manager maintains fleet vehicle information, including adding new vehicles to the fleet vehicle database; removing old or discontinued vehicles from the fleet vehicle database; managing vehicle capability profiles, wherein the vehicle capability profiles include sensor configuration, available sensor types, available motor vehicle types, and maximum available resources; and updating new vehicle features and capabilities.

[0043] Option 9. The continuous verification system for motor vehicles according to Option 1, wherein:

[0044] The vehicle fleet comprises multiple motor vehicles of various brands and models operating anywhere in the world; and

[0045] The at least one customer includes service users, suppliers, original equipment manufacturers (OEMs), and government agencies.

[0046] Option 10. The continuous verification system for motor vehicles according to Option 1 includes a simulator that communicates with the verification manager. The simulator operates to perform a profiling analysis on the plurality of verification tasks before they are deployed to confirm whether vehicles that may request the plurality of verification tasks have the capability to perform the verification analysis.

[0047] Option 11. A continuous verification system for motor vehicles, comprising:

[0048] The backend collects data from and wirelessly communicates with at least one motor vehicle, and also wirelessly communicates with at least one client.

[0049] A vehicle module is provided for mounting on the at least one motor vehicle and for performing verification analysis;

[0050] A verification manager that generates multiple verification tasks for the verification analysis based on user definitions or the expected output of multiple verification tasks and the availability of the at least one motor vehicle.

[0051] A simulator that performs a procedural analysis of the verification task prior to its deployment to confirm whether at least one motor vehicle requesting the plurality of verification tasks has the capability to perform the verification analysis; and

[0052] A client module, which is remote relative to the at least one motor vehicle, has an interface item provided by at least one client application seeking to perform the verification analysis.

[0053] Option 12. The continuous verification system for motor vehicles according to Option 11, wherein the at least one motor vehicle defines a plurality of motor vehicles in a vehicle fleet.

[0054] Option 13. The continuous verification system for motor vehicles according to Option 12 further includes a fleet-based verification module, or provides the fleet-based verification module at the backend or based on the cloud to communicate with the verification manager, wherein the fleet-based verification module manages data defining the configuration and capability profiles of multiple motor vehicles in the vehicle fleet.

[0055] Option 14. The continuous verification system for motor vehicles according to Option 12, comprising:

[0056] A wireless communication module receives the plurality of verification tasks, multiple queries, and multiple commands, schedules the execution of the plurality of verification tasks based on a predefined priority plan, and interacts with the vehicle user; and

[0057] A wireless client communication downlink, wherein in response to the plurality of verification tasks, data including vehicle operating status, verification operation progress, verification results and costs are returned from the backend via the wireless client communication downlink.

[0058] Option 15. The continuous verification system for motor vehicles according to Option 14, comprising:

[0059] The task manager communicates with the wireless communication module;

[0060] The job manager communicates with the task manager; and

[0061] A job actuator, which is responsible for performing tasks and controlling multiple sensor gateways and vehicle systems, so that data will be received by the job actuator.

[0062] Option 16. The continuous verification system for motor vehicles according to Option 15 includes a configuration file manager that communicates with the task manager and the job executor and manages vehicle capability configuration files, the vehicle capability configuration files including sensor configuration, available sensor types, types of available motor vehicles among multiple motor vehicles in the vehicle fleet, and maximum available resources, the configuration file manager also managing runtime configuration files including resource usage and task and job statistics.

[0063] Option 17. The continuous verification system for motor vehicles according to Option 16, comprising:

[0064] A first gateway is defined to communicate with the job executor and the job manager, the sensor gateway communicating with multiple on-board sensors of multiple vehicles in the vehicle fleet, the multiple on-board sensors generating sensing signals in response to the operating environment of the multiple vehicles in the vehicle fleet; and

[0065] A second gateway that communicates with the job manager defines the vehicle system gateway.

[0066] Option 18. A method for performing continuous verification of a vehicle fleet, comprising:

[0067] Data is collected from the vehicle fleet via a backend and communicates wirelessly with the vehicle fleet; the backend also communicates wirelessly with at least one customer.

[0068] Perform onboard vehicle verification analysis using the vehicle module provided for each of the multiple vehicles in the vehicle fleet;

[0069] Use or provide a fleet-based verification module at the backend or based on the cloud to manage data defining the configuration and capability profiles of multiple motor vehicles in the vehicle fleet;

[0070] The verification task is generated using the verification manager based on the user's definition or the expected output of the verification task from the verification analysis and the fleet vehicle availability; and

[0071] The verification analysis is performed by at least one client seeking to perform the analysis via a client module application interface project, the client module being remote relative to the multiple vehicles in the vehicle fleet.

[0072] Option 19. The method according to Option 18 further includes maintaining fleet vehicle information using a fleet vehicle manager that communicates with a fleet vehicle database and communication equipment, including adding new vehicles to the fleet vehicle database; removing old or discontinued vehicles from the fleet vehicle database; managing the capability profile, wherein the capability profile includes sensor configuration, available sensor types, available vehicle types, and maximum available resources; and updating new vehicle features and capabilities.

[0073] Option 20. The method according to Option 18 further includes using a simulator communicating with the verification manager to perform a profiling analysis on the verification task before the verification task is deployed, to confirm whether at least one of a plurality of motor vehicles that may request the verification task has the capability to perform the verification analysis.

[0074] Other application areas will become apparent from the description provided herein. It should be understood that the descriptions and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0075] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0076] Figure 1 It is a schematic representation of the principle characteristics of a continuous verification system for motor vehicles based on an exemplary aspect;

[0077] Figure 2 It indicates that it is provided for vehicle use. Figure 1 A system diagram of individual vehicle module components on a motor vehicle system;

[0078] Figure 3 It is used for Figure 1 A system diagram of the components of the fleet-based verification module; and

[0079] Figure 4 It is marked Figure 1 A system diagram showing the components of the client module of the system. Detailed Implementation

[0080] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses.

[0081] refer to Figure 1The continuous vehicle verification system 10 includes a backend 12, which may include remote computers, servers, cloud-based computing systems, etc. The backend 12 collects data from a vehicle fleet 14, which may contain multiple vehicles 16, 18, and 20 of different brands and models, operating anywhere in the world. The backend 12 wirelessly communicates with the vehicle fleet 14 via a wireless fleet communication uplink 22, which forwards tasks, jobs, and queries to the vehicle fleet 14. Data, including a set of vehicle profiles, verification progress, and collected data, is returned from the vehicle fleet 14 to the backend 12 via a wireless fleet communication downlink 24.

[0082] Backend 12 also communicates wirelessly with one or more clients 26. Clients 26 may include service users, suppliers, original equipment manufacturers (OEMs), government agencies, etc. Clients 26 wirelessly upload data and requests for vehicle fleet verification data to backend 12 via wireless client communication uplink 28. The uploaded data includes verification requests, task definitions, and job queries. In response to verification requests, data including vehicle operating status, verification job progress, verification results, and costs are returned from backend 12 via wireless client communication downlink 30. Clients 26 can thus select at least one or more features of the vehicles in the vehicle fleet 14 that they wish to verify via the verification request, such as, but not limited to, autonomous vehicle handling, braking, steering, control, etc., and can also request fleet verification data already stored in backend 12.

[0083] refer to Figure 2 And refer again Figure 1 System diagram 32 illustrates components of a vehicle module 34 provided for installation on each of the multiple motor vehicles 16, 18, 20 in a vehicle fleet 14, for performing on-board vehicle verification. Vehicle module 34 includes a wireless communication component 36 that defines a wireless communication module that manages communication between backend 12 and the various vehicle systems of one or more vehicles in the vehicle fleet 14. Wireless communication component 36 receives multiple tasks, commands, and queries from backend 12 and forwards verification results and a set of vehicle profiles from the vehicle fleet 14 to backend 12.

[0084] The task manager 38, which communicates with the wireless communication component 36, manages multiple tasks, queries, and commands, schedules their execution based on predefined priorities, and interacts with vehicle users as needed. The task manager 38 also forwards the results of verification requests, requested and generated data, and a set of vehicle profiles to the backend 12. The task manager 38 communicates with the human-machine interface (HMI) 40 and is also responsible for handling overrun signals 42 from any operator, user, or passenger in one or more of the vehicles in the vehicle fleet 14. For example, an overrun signal 42 might be generated if a vehicle operator deems communication with the backend 12 undesirable during driving conditions such as peak traffic hours or inclement weather.

[0085] The job manager 44, communicating with the task manager 38 and the job executor 46, controls the sensor gateway and vehicle system, ensuring that relevant data is received by the job executor 46 responsible for performing the job. The job manager 44 also configures system resources and manages the run-time of requested jobs. The configuration file manager 48, also communicating with the task manager 38 and the job executor 46, manages vehicle capability configuration files, including sensor configuration, available sensor types, available vehicle types, and maximum available resources. The configuration file manager 48 manages runtime configuration files, including resource usage and task and job statistics.

[0086] Vehicle module 34 also includes various modules, including a first gateway defining a sensor gateway 50 that communicates with the operation actuator 46 and the operation manager 44. The sensor gateway 50 communicates with multiple onboard sensors 52 that provide sensing signals for the operating environment of the motor vehicle. Each module of vehicle module 34 also includes a second gateway defining a vehicle system gateway 54 that communicates with the operation manager 44. The vehicle system gateway 54 transmits relevant data to be verified received from multiple vehicle systems 56, such as, but not limited to, headlights, windshield wipers, door locks, etc. Vehicle systems 56 include data to be verified, such as the data mentioned above, as well as data that does not require verification, such as vehicle power control, battery control, and parking characteristics. A storage unit 58, communicating with the operation actuator 46 and the task manager 38, stores copies of the verification data. The storage unit 58 may be located in the various modules of vehicle module 34 or in the backend 12.

[0087] Uploaded data packet 60 is transferred from wireless communication component 36 to task manager 38. For example, downloaded data packet 62, including verification results, is transferred from task manager 38 to wireless communication component 36.

[0088] refer to Figure 3 And refer again Figure 1 and Figure 2System diagram 64 illustrates the components of the fleet-based verification module 66. The fleet-based verification module 66 is provided either at the backend 12 or via the cloud. The fleet-based verification module 66 manages data defining the configuration and capabilities of each vehicle in the vehicle fleet 14, including sensor configuration, available sensor types, available vehicle types, and maximum available resources. The fleet-based verification module 66 manages the jobs being processed by each vehicle and the tasks scheduled for the vehicles, as well as the verification data collected from each vehicle. The fleet-based verification module 66 also creates, dispatches, suspends, or cancels tasks for available vehicles in the vehicle fleet 14.

[0089] The fleet-based verification module 66 includes a communication device 68 for communicating with individual vehicles of the vehicle fleet 14 that have the target characteristics for verification. The communication device 68 provides input to and receives data from the verification manager 70. The verification manager 70 generates multiple verification tasks based on user-defined or desired verification task outputs and fleet vehicle availability. The fleet vehicle manager 72, communicating with the fleet vehicle database 74 and the communication device 68, maintains fleet vehicle information, such as adding new vehicles to the fleet vehicle database 74 due to vehicle sales and removing older or discontinued vehicles. The fleet vehicle manager 72 also manages vehicle capability information and updates new vehicle characteristics and capabilities. The fleet vehicle manager 72 also provides search or query functions to find capable vehicles in the fleet to support the requested verification task. The fleet vehicle manager 72 communicates with the program, manufacturing, and sales interface 78 to retrieve data for updating the fleet vehicle database 74.

[0090] Security manager 79, communicating with communication device 68, ensures that user-defined tasks and the algorithms required to perform verification operations are secure and reliable. Security manager 79 communicates with user manager 80 and user interface 82. User manager 80 manages user profiles, user-requested tasks, verification data, and task costs. For example, user manager 80 identifies whether a user has already paid for the requested service or services requested in the past, identifies potential users who have registered to use the system, and monitors the user list to maintain which users retain access to the system. User interface 82 selects data from fleet vehicle database 74 and fleet vehicle manager 72 to identify the data required for the execution of verification requests.

[0091] Cost center 84 communicates with verification manager 70 and user manager 80 to identify whether inventives are available for application to user analysis and verification requests; and when receiving and verifying tasks, cost center 84 identifies the cost of collecting and analyzing available data in the system. Simulator 86, communicating with verification manager 70, allows for a profiling analysis of tasks before deployment to confirm whether vehicles that might request tasks have the capability to perform verification work. Therefore, tasks cannot be forwarded to requesting vehicles that have failed simulation tests, thus saving computation time and cost. All data used and all verification requests made to or by the system can be stored in storage device 88, which may be local or cloud-based.

[0092] refer to Figure 4 And refer again Figures 1 to 3 System diagram 90 identifies the elements of client module 92, which is remote relative to the vehicles of vehicle fleet 14 and represents the interface item for a system client application seeking to perform verification analysis. Client module 92 includes client interface 94, which monitors and controls client requests output to fleet-based verification module 66 and verification data returned to the client. Client interface 94 communicates with account management component 96, verification request management component 98, and fleet management component 100. Account management component 96 manages the status of individual client or customer accounts and includes dedicated storage device 102 containing algorithms in the form of tasks to be verified and any relevant reference algorithms. Verification request management component 98 permits the client or customer to generate estimates of the computational resources required to respond to a client request for data. Fleet management component 100 allows the client or customer to identify and select one or more vehicles available in vehicle fleet 14, which have the necessary sensors and computing capabilities for the desired verification task.

[0093] The verification request management component 98 communicates with the verification process monitor 104, verification process manager 106, verification strategy component 108, and analysis and visualization component 110. The verification process monitor 104 provides the customer with monitoring capabilities to track the real-time progress of verification operations and allows the customer to evaluate and verify verification progress and data collected during the process. The verification process manager 106 allows the customer to manage the verification process, such as restarting, pausing, resuming, and terminating the verification process. The verification strategy component 108 allows the customer to define verification metrics and deploy targets and reference algorithms and matrices to selected vehicles or multiple vehicles. The analysis and visualization component 110 allows the customer to calculate the total cost of verification tasks; manage and visualize different scenarios before sending verification task requests, including identifying how many vehicles and which vehicles are included in the task request; and perform analysis and visualization using the captured data.

[0094] The disclosed continuous verification system for motor vehicles provides an architecture and system that allows OEMs to use large-scale vehicle production as a platform to assist in the verification of AV / ADAS features.

[0095] The disclosed continuous validation system for motor vehicles offers several advantages. These advantages include providing validation as a service for performing on-demand tasks. The system design supports collaboration between vehicles, backend servers, and validation experts. Customer-driven, demand-based selection from a fleet of vehicles generates a large volume of data, rather than performing validation based on a small number of test vehicles.

[0096] Continuous validation allows for the validation of features during each vehicle run, and thus covers more scenarios that are difficult to build or test during the testing phase. The system disclosed herein allows for the validation of one or more features already in production for further performance tuning and improvements, and also provides validation for new features not yet in production vehicles.

[0097] As used herein, a controller is a non-general-purpose electronic control device that has a pre-programmed digital computer or processor, or a memory or non-transitory computer-readable medium for storing data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver or input / output port. Computer-readable media includes any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, compact disc (CD), digital video disc (DVD), or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical, or other communication links that transmit transient electrical signals or other signals. Non-transitory computer-readable media may also include media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical discs or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code. The processor is configured to execute code or instructions. In the case where the vehicles in vehicle fleet 14 are automobiles, autonomous vehicles, electric vehicles, etc., the controller or module may be a dedicated Wi-Fi controller or an engine control module, transmission control module, body control module, infotainment control module, etc. Wireless transmission equipment may include a transceiver configured to communicate wirelessly with hotspots or the cloud using, for example, Wi-Fi protocols under IEEE 802.11x.

[0098] Motor vehicles also include one or more application programs. An application program is a software program configured to perform a specific function or set of functions. An application program may include one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, related data, or portions thereof suitable for implementation in appropriate computer-readable program code. The application program may be stored within the module's memory or in additional or separate memory.

[0099] The description in this disclosure is merely exemplary in nature, and variations thereof without departing from the spirit and scope of this disclosure are intended to fall within its scope. Such variations are not considered to depart from the spirit and scope of this disclosure.

Claims

1. A continuous verification system for motor vehicles, comprising: The backend collects data from and communicates wirelessly with the vehicle fleet, and also communicates wirelessly with at least one customer. A vehicle module, which is provided to be mounted on each of the multiple motor vehicles in the vehicle fleet and to perform on-board vehicle verification analysis; A fleet-based verification module, which is provided either at the backend or in the cloud, manages data defining the configuration and capability profiles of multiple vehicles in the vehicle fleet. A validation manager that generates multiple validation tasks based on user-defined or expected outputs of multiple validation tasks included in a validation analysis and fleet vehicle availability; and A client module, which is remote relative to multiple vehicles in the vehicle fleet, has an interface item consisting of at least one client application seeking to perform the verification analysis. A simulator that communicates with the verification manager runs to perform a profiling analysis of the plurality of verification tasks before they are deployed, in order to confirm whether vehicles that may request the plurality of verification tasks have the capability to perform the verification analysis.

2. The continuous verification system for motor vehicles according to claim 1, wherein, The vehicle module includes: A wireless communication component that manages communication between the backend and the various vehicle systems of the plurality of motor vehicles; and The task manager receives the multiple verification tasks, multiple queries, and multiple commands, schedules the execution of the multiple verification tasks, multiple queries, and multiple commands based on a predefined priority plan, interacts with the vehicle user, and forwards the results of the verification task requests, the requested and generated data, and a set of vehicle configuration files to the backend.

3. The continuous verification system for motor vehicles according to claim 2 further includes a human-machine interface (HMI), wherein the task manager also communicates with the HMI and is responsible for processing override signals received via the HMI from one or more users of the motor vehicles in the vehicle fleet.

4. The continuous verification system for motor vehicles according to claim 2, wherein, The task manager communicates with the wireless communication component; and also includes a verification request management component that authorizes the at least one client to generate an estimate of the required computing resources necessary to respond to a client request for data.

5. The continuous verification system for motor vehicles according to claim 2, wherein, The wireless communication component receives the multiple verification tasks, multiple queries, and multiple commands from the backend, and forwards the verification analysis results and the set of vehicle configuration files from the vehicle fleet to the backend.

6. The continuous verification system for motor vehicles according to claim 1, comprising: A wireless client communication uplink, wherein at least one client communicates with the wireless client communication uplink to wirelessly upload data and requests for vehicle fleet verification data to the backend via the wireless client communication uplink, the uploaded data and requests including multiple verification requests, task definitions and job queries; and The wireless client communication downlink includes data on vehicle operating status, verification operation progress, verification results, and costs, which are returned to the backend via the wireless client communication downlink.

7. The continuous verification system for motor vehicles according to claim 1, wherein, The client module includes a client interface for monitoring and controlling requests sent from the at least one client to the fleet-based verification module and verification data returned to the at least one client.

8. The continuous verification system for motor vehicles according to claim 1 further includes a fleet vehicle manager, which communicates with a fleet vehicle database and communication equipment, and maintains fleet vehicle information, including adding new vehicles to the fleet vehicle database; removing old or discontinued vehicles from the fleet vehicle database; managing vehicle capability profiles, wherein the vehicle capability profiles include sensor configuration, available sensor types, available motor vehicle types, and maximum available resources; and updating new vehicle features and capabilities.

9. The continuous verification system for motor vehicles according to claim 1, wherein: The vehicle fleet comprises multiple motor vehicles of various brands and models operating anywhere in the world; and The at least one customer includes service users, suppliers, original equipment manufacturers (OEMs), and government agencies.

10. A continuous verification system for motor vehicles, comprising: The backend collects data from and wirelessly communicates with at least one motor vehicle, and also wirelessly communicates with at least one client. A vehicle module is provided for mounting on the at least one motor vehicle and for performing verification analysis; A verification manager that generates multiple verification tasks for the verification analysis based on user definitions or the expected output of multiple verification tasks and the availability of the at least one motor vehicle. A simulator that performs a procedural analysis of the verification task before the verification task is deployed to confirm whether at least one motor vehicle requesting the plurality of verification tasks has the capability to perform the verification analysis. and A client module, which is remote relative to the at least one motor vehicle, has an interface item provided by at least one client application seeking to perform the verification analysis.

11. The continuous verification system for motor vehicles according to claim 10, wherein, The at least one motor vehicle defines a fleet of multiple motor vehicles.

12. The continuous verification system for motor vehicles according to claim 11 further includes a fleet-based verification module, or the fleet-based verification module provided at the backend or via the cloud communicates with a verification manager, the fleet-based verification module managing data defining the configuration and capability profiles of multiple motor vehicles in the vehicle fleet.

13. The continuous verification system for motor vehicles according to claim 11, comprising: The wireless communication module receives the plurality of verification tasks, the plurality of queries, and the plurality of commands, and schedules the execution of the plurality of verification tasks based on a predefined priority plan, and interacts with the vehicle user; and A wireless client communication downlink, wherein in response to the plurality of verification tasks, data including vehicle operating status, verification operation progress, verification results and costs are returned from the backend via the wireless client communication downlink.

14. The continuous verification system for motor vehicles according to claim 13, comprising: The task manager communicates with the wireless communication module; The job manager communicates with the task manager; and A job actuator, which is responsible for performing tasks and controlling multiple sensor gateways and vehicle systems, so that data will be received by the job actuator.

15. The continuous verification system for motor vehicles according to claim 14, comprising a configuration file manager that communicates with the task manager and the job executor and manages vehicle capability configuration files, the vehicle capability configuration files including sensor configuration, available sensor types, types of available motor vehicles among a plurality of motor vehicles in the vehicle fleet, and maximum available resources, the configuration file manager also managing runtime configuration files including resource usage and task and job statistics.

16. The continuous verification system for motor vehicles according to claim 15, comprising: A first gateway is defined to communicate with the job executor and the job manager, the sensor gateway communicating with multiple on-board sensors of multiple vehicles in the vehicle fleet, the multiple on-board sensors generating sensing signals in response to the operating environment of the multiple vehicles in the vehicle fleet; and A second gateway that communicates with the job manager defines the vehicle system gateway.

17. A method for performing continuous verification of a vehicle fleet, comprising: Data is collected from the vehicle fleet via a backend and communicates wirelessly with the vehicle fleet; the backend also communicates wirelessly with at least one customer. Perform onboard vehicle verification analysis using the vehicle module provided for each of the multiple vehicles in the vehicle fleet; Use or provide a fleet-based verification module at the backend or based on the cloud to manage data defining the configuration and capability profiles of multiple motor vehicles in the vehicle fleet; The verification task is generated using the verification manager based on the user's definition or the expected output of the verification task from the verification analysis and the fleet vehicle availability. and The verification analysis is performed by at least one client seeking to perform the analysis via a client module application interface project, the client module being remote relative to multiple vehicles in the vehicle fleet; Using a simulator that communicates with the verification manager, a profiling analysis is performed on the verification task before it is deployed to confirm whether at least one of the multiple motor vehicles that may request the verification task has the capability to perform the verification analysis.

18. The method of claim 17, further comprising maintaining fleet vehicle information using a fleet vehicle manager communicating with a fleet vehicle database and communication equipment, including adding new vehicles to the fleet vehicle database; removing old or discontinued vehicles from the fleet vehicle database; managing the capability profile, wherein the capability profile includes sensor configuration, available sensor types, available vehicle types, and maximum available resources; and updating new vehicle features and capabilities.

Images