Neural Network Validation System

Abstract

This invention relates to a neural network verification system. The system includes a computer comprising a processor and a memory. The memory includes instructions that cause the processor to: receive unlabeled sensor data at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data; receive unlabeled sensor data at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode; the second neural network, unlike the first neural network, compares the output generated by the first neural network with the output generated by the second neural network, and generates an alarm when the difference between the output generated by the first neural network and the output generated by the second neural network exceeds a predetermined comparison threshold.

Description

Technical Field

[0001] This disclosure relates to using outputs from multiple other neural network models to validate (e.g., cross-check) neural network outputs. Background Technology

[0002] Deep neural networks (DNNs) can be used to perform many image understanding tasks, including classification, segmentation, and adding descriptive text. Typically, DNNs require a large number of training images (tens of thousands to millions). Furthermore, these training images usually need to be annotated, such as labeled, for training and prediction purposes. Summary of the Invention

[0003] A system includes a computer comprising a processor and a memory. The memory includes instructions that program the processor to: receive unlabeled sensor data at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data; receive unlabeled sensor data at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode; the second neural network, unlike the first neural network, compares the output generated by the first neural network with the output generated by the second neural network, and generates an alarm when the difference between the output generated by the first neural network and the output generated by the second neural network exceeds a predetermined comparison threshold.

[0004] Among other features, the processor is further programmed to receive a selection for switching between verification mode and feature mode.

[0005] Among other features, the processor is further programmed to operate at least one vehicle actuator based on the output generated by the first neural network during the feature pattern.

[0006] Among the other features, select to transfer from the server.

[0007] Among other features, the option is to transmit data from the vehicle's electronic control unit.

[0008] Among other features, a first neural network is trained using a first dataset, and a second neural network is trained using a second dataset, wherein the second dataset is different from the first dataset.

[0009] Among other features, the processor is further programmed to prevent the output generated by the first neural network from being used to operate the vehicle during verification mode.

[0010] Among other characteristics, unlabeled sensor data includes sensor data collected by the fleet.

[0011] The vehicle includes a system. This system includes a computer, which includes a processor and a memory. The memory includes instructions that cause the processor to: receive unlabeled sensor data at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data; receive unlabeled sensor data at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode; the second neural network, unlike the first neural network, compares the output generated by the first neural network with the output generated by the second neural network, and generates an alarm when the difference between the output generated by the first neural network and the output generated by the second neural network exceeds a predetermined comparison threshold.

[0012] Among other features, the processor is further programmed to receive a selection for switching between verification mode and feature mode.

[0013] Among other features, the processor is further programmed to operate at least one vehicle actuator of the vehicle based on the output generated by the first neural network during the feature mode.

[0014] Among the other features, select to transfer from the server.

[0015] Among other features, the option is to transmit data from the vehicle's electronic control unit.

[0016] Among other features, a first neural network is trained using a first dataset, and a second neural network is trained using a second dataset, which is different from the first dataset.

[0017] Among other features, the processor is further programmed to prevent the output generated by the first neural network from being used to operate the vehicle during verification mode.

[0018] Among other characteristics, unlabeled sensor data includes sensor data collected by the fleet.

[0019] A method includes receiving unlabeled sensor data at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data, receiving unlabeled sensor data at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode, the second neural network being different from the first neural network, comparing the output generated by the first neural network with the output generated by the second neural network, and generating an alarm when the difference between the output generated by the first neural network and the output generated by the second neural network is greater than a predetermined comparison threshold.

[0020] Among other features, the method includes receiving a selection for switching between a verification mode and a feature mode.

[0021] Among other features, the method includes operating at least one vehicle actuator based on the output generated by the first neural network during the feature pattern.

[0022] Among the other features, select to transfer from the server.

[0023] This disclosure provides the following technical solutions:

[0024] 1. A system including a computer, the computer including a processor and a memory, the memory including instructions that program the processor to:

[0025] Unlabeled sensor data is received at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data;

[0026] The unlabeled sensor data is received at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode, and the second neural network is different from the first neural network;

[0027] The output generated by the first neural network is compared with the output of the second neural network; and

[0028] An alarm is generated when the difference between the output generated by the first neural network and the output generated by the second neural network exceeds a predetermined comparison threshold.

[0029] 2. The system according to technical solution 1, wherein the processor is further programmed to receive a selection for switching between verification mode and feature mode.

[0030] 3. The system according to claim 2, wherein the processor is further programmed to operate at least one vehicle actuator based on the output generated by the first neural network during a feature pattern.

[0031] 4. The system according to technical solution 2, wherein the selection is transmitted from the server.

[0032] 5. The system according to technical solution 2, wherein the selection is transmitted from the vehicle's electronic controller unit.

[0033] 6. The system according to technical solution 1, wherein a first dataset is used to train the first neural network, and a second dataset is used to train the second neural network, wherein the second dataset is different from the first dataset.

[0034] 7. The system according to claim 1, wherein the processor is further programmed to prevent the output generated by the first neural network from being used to operate the vehicle during the verification mode.

[0035] 8. The system according to technical solution 1, wherein the unlabeled sensor data includes sensor data collected by the fleet.

[0036] 9. A vehicle including a system, the system including a computer, the computer including a processor and a memory, the memory including instructions that program the processor to:

[0037] Unlabeled sensor data is received at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data;

[0038] The unlabeled sensor data is received at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode, and the second neural network is different from the first neural network;

[0039] The output generated by the first neural network is compared with the output of the second neural network; and

[0040] An alarm is generated when the difference between the output generated by the first neural network and the output generated by the second neural network exceeds a predetermined comparison threshold.

[0041] 10. The vehicle according to claim 9, wherein the processor is further programmed to receive a selection for switching between a verification mode and a feature mode.

[0042] 11. The vehicle according to claim 10, wherein the processor is further programmed to operate at least one vehicle actuator of the vehicle based on the output generated by the first neural network during a feature pattern.

[0043] 12. The vehicle according to technical solution 10, wherein the selection is transmitted from the server.

[0044] 13. The system according to claim 10, wherein the selection is transmitted from the vehicle's electronic controller unit.

[0045] 14. The system according to technical solution 9, wherein a first dataset is used to train the first neural network, and a second dataset is used to train the second neural network, wherein the second dataset is different from the first dataset.

[0046] 15. The system according to claim 9, wherein the processor is further programmed to prevent the output generated by the first neural network from being used to operate the vehicle during the verification mode.

[0047] 16. The system according to claim 9, wherein the unlabeled sensor data includes sensor data collected by the fleet.

[0048] 17. A method comprising:

[0049] Unlabeled sensor data is received at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data;

[0050] The unlabeled sensor data is received at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode, and the second neural network is different from the first neural network;

[0051] The output generated by the first neural network is compared with the output of the second neural network; and

[0052] An alarm is generated when the difference between the output generated by the first neural network and the output generated by the second neural network exceeds a predetermined comparison threshold.

[0053] 18. The method according to technical solution 17 further includes receiving a selection for switching between verification mode and feature mode.

[0054] 19. The method according to claim 18 further includes operating at least one vehicle actuator based on the output generated by the first neural network during a feature pattern.

[0055] 20. The system according to technical solution 18, wherein the selection is transmitted from the server.

[0056] Further areas of applicability should become apparent from the description provided herein. It should be understood that the descriptions and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0057] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0058] Figure 1 It is a block diagram of a vehicle system including a verification network used to compare the output generated by a first neural network with the outputs generated by multiple neural networks;

[0059] Figure 2 This is a block diagram of the sample server within the system;

[0060] Figure 3 This is a diagram illustrating an example neural network;

[0061] Figure 4 Here is a block diagram of an example validation network; and

[0062] Figure 5 This is a flowchart illustrating an example process for verifying the output generated by a neural network. Detailed Implementation

[0063] The following description is exemplary in nature and is not intended to limit this disclosure, application, or use.

[0064] Typically, standard deep neural networks (DNNs) are pre-trained using labeled training datasets. These DNNs can be validated during testing by comparing the model's output to ground truth benchmarks. However, in real-world testing scenarios, obtaining ground truth benchmark data can be difficult. Furthermore, testing DNNs can reveal the root causes that require further analysis to identify incorrect DNN outputs.

[0065] This disclosure discloses a neural network validation system in which the output generated by a neural network is compared with the output generated by a validation neural network. The validation neural network can be trained on different datasets, which may be a subset of observations with different biases from the real-world baseline distribution. For example, the validation neural network may include an architecture different from that of the neural network of interest.

[0066] Figure 1 This is a block diagram of an example vehicle system 100. System 100 includes a vehicle 105, which is a land vehicle such as a car, truck, etc. Vehicle 105 includes a computer 110, vehicle sensors 115, actuators 120 for actuating various vehicle components 125, and a vehicle communication module 130. The communication module 130 allows the computer 110 to communicate with a server 145 via a network 135.

[0067] Computer 110 includes a processor and memory. The memory includes one or more forms of computer-readable medium and stores instructions executable by computer 110 to perform various operations including those disclosed herein.

[0068] Computer 110 can operate vehicle 105 in autonomous, semi-autonomous, or non-autonomous (manual) modes. For the purposes of this disclosure, autonomous mode is defined as a mode in which each of the propulsion, braking, and steering of vehicle 105 is controlled by computer 110; in semi-autonomous mode, computer 110 controls one or both of the propulsion, braking, and steering of vehicle 105; and in non-autonomous mode, a human operator controls each of the propulsion, braking, and steering of vehicle 105.

[0069] Computer 110 may include programming to operate vehicle 105 braking, propulsion (e.g., controlling vehicle acceleration by controlling one or more of an internal combustion engine, electric motor, hybrid engine, etc.), steering, climate control, interior and / or exterior lights, etc., and to determine whether and when computer 110, rather than a human operator, controls such operations. Furthermore, computer 110 may be programmed to determine whether and when a human operator controls such operations.

[0070] Computer 110 may include more than one processor, or be communicatively coupled to more than one processor, for example via vehicle 105 communication module 130 as further described below. This more than one processor may be included, for example, in an electronic control unit (ECU) or similar component included in vehicle 105, for monitoring and / or controlling various vehicle components 125, such as powertrain controllers, brake controllers, steering controllers, etc. Furthermore, computer 110 may communicate with a navigation system using a Global Positioning System (GPS) via vehicle 105 communication module 130. For example, computer 110 may request and receive location data of vehicle 105. The location data may be in a known form, such as geographic coordinates (latitude and longitude coordinates).

[0071] Computer 110 is typically arranged to communicate on vehicle 105 communication module 130 and also with wired and / or wireless networks within vehicle 105, such as buses in vehicle 105, such as controller area networks (CAN), and / or other wired and / or wireless mechanisms.

[0072] Via the vehicle 105 communication network, the computer 110 can transmit and / or receive messages from various devices within the vehicle 105, such as vehicle sensors 115, actuators 120, vehicle components 125, human-machine interfaces (HMIs), etc. Alternatively or additionally, where the computer 110 actually comprises multiple devices, the vehicle 105 communication network can be used for communication between devices represented herein as computer 110. Furthermore, as described below, various controllers and / or vehicle sensors 115 can provide data to the computer 110. The vehicle 105 communication network may include one or more gateway modules that provide interoperability between various networks and devices within the vehicle 105, such as protocol converters, impedance matching devices, and rate converters.

[0073] Vehicle sensor 115 may include a wide variety of devices, such as those known for providing data to computer 110. For example, vehicle sensor 115 may include one or more light-detecting and ranging (lidar) sensors 115, etc., mounted on the top of vehicle 105, behind the windshield of vehicle 105, around vehicle 105, etc., providing the relative position, size, shape, and / or condition of objects around vehicle 105. As another example, one or more radar sensors 115 fixed to the bumper of vehicle 105 may provide data to provide and range the speed and distance of objects (possibly including a second vehicle 106) relative to the position of vehicle 105. Vehicle sensor 115 may further include one or more camera sensors 115, such as forward-view camera sensors, side-view camera sensors, rear-view camera sensors, etc., providing images of the field of view from inside and / or outside vehicle 105.

[0074] The actuator 120 of vehicle 105 is implemented via circuitry, chips, motors, or other electronic and / or mechanical components that can actuate various vehicle subsystems according to known and appropriate control signals. The actuator 120 can be used to control components 125, including braking, acceleration, and steering of vehicle 105.

[0075] In the context of this disclosure, vehicle component 125 is one or more hardware components adapted to perform mechanical or electromechanical functions or operations—such as moving vehicle 105, slowing or stopping vehicle 105, steering vehicle 105, etc. Non-limiting examples of component 125 include propulsion components (which include, for example, internal combustion engines and / or electric motors), transmission components, steering components (e.g., may include one or more of a steering wheel, steering rack, etc.), braking components (described below), parking assist components, adaptive cruise control components, adaptive steering components, movable seats, etc.

[0076] Furthermore, computer 110 can be configured to communicate with devices outside vehicle 105 via vehicle-to-vehicle communication module or interface 130, for example, via vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2X) wireless communication to another vehicle, to a remote server 145 (typically via network 135). Module 130 may include one or more mechanisms through which computer 110 can communicate, including wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or any desired combination of topologies when multiple communication mechanisms are utilized). Exemplary communications provided via module 130 include cellular, Bluetooth®, IEEE 802.11, Private Short Range Communication (DSRC), and / or wide area network (WAN), including the Internet, thereby providing data communication services.

[0077] Network 135 can be one or more of various wired or wireless communication mechanisms, including wired (e.g., cable and fiber optic) and / or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or any desired combination when multiple communication mechanisms are utilized). Exemplary communication networks include wireless communication networks (e.g., using Bluetooth, Bluetooth Low Energy (BLE), IEEE 802.11, vehicle-to-vehicle (V2V) communication such as Dedicated Short Range Communication (DSRC), etc.), local area networks (LANs), and / or wide area networks (WANs), including the Internet, thereby providing data communication services.

[0078] Computer 110 can receive and analyze data from sensor 115 substantially continuously, periodically, and / or when instructed by server 145. Furthermore, object classification or identification techniques can be used, for example, within computer 110 based on data from lidar sensor 115, camera sensor 115, etc., to identify the type and physical characteristics of objects such as vehicles, people, rocks, potholes, bicycles, motorcycles, etc.

[0079] Figure 2 This is a block diagram of example server 145. Server 145 includes computer 235 and communication module 240. Computer 235 includes a processor and memory. Memory includes one or more forms of computer-readable medium and stores instructions executable by computer 235 to perform various operations including those disclosed herein. Communication module 240 allows computer 235 to communicate with other devices, such as vehicle 105.

[0080] Figure 3 This is a diagram illustrating an example deep neural network (DNN) 300 used in this paper. The DNN 300 includes multiple nodes 305, and the nodes 305 are arranged such that the DNN 300 includes an input layer, one or more hidden layers, and an output layer. Each layer of the DNN 300 may include multiple nodes 305. Although Figure 3 The diagram shows three (3) hidden layers, but it should be understood that the DNN300 may include additional or fewer hidden layers. The input and output layers may also include more than one (1) node 305.

[0081] Nodes 305 are sometimes referred to as artificial neurons 305 because they are designed to mimic biological (e.g., human) neurons. The input set to each neuron 305 (indicated by arrows) is each multiplied by a corresponding weight. The weighted inputs can then be summed in an input function to provide a net input (possibly adjusted for bias). This net input can then be provided to an activation function, which in turn provides the output to the connected neurons 305. Activation functions can be a wide variety of suitable functions, typically chosen based on empirical analysis. For example, through... Figure 3 As illustrated by the arrow in the diagram, the output of neuron 305 can then be provided to be included in the input set of one or more neurons 305 in the next layer.

[0082] The DNN 300 can be trained to accept data as input and generate output based on that input. In one example, the DNN 300 can be trained using ground truth data, i.e., data about real-world conditions or states. For example, the DNN 300 can be trained by a processor using ground truth data or updated using additional data. For example, the weights can be initialized using a Gaussian distribution, and the bias of each node 305 can be set to zero. Training the DNN 300 can include updating the weights and biases via suitable techniques, such as backpropagation with optimization. Ground truth data can include, but is not limited to, data on objects within a specified image or data on specified physical parameters, such as angle, velocity, distance, color, hue, or the angle of one object relative to another. For example, ground truth data can be data representing objects and object labels.

[0083] Machine learning services, such as those based on recurrent neural networks (RNNs), convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, or gated recurrent units (GRUs), can be implemented using the DNN 300 described in this disclosure. In one example, service-related content or other information (such as words, sentences, images, videos, or other such content / information) can be transformed into vector representations.

[0084] Figure 4 This is a diagram illustrating an example verification network 400 used to compare the output generated by a neural network 405 (e.g., a first neural network) with the output generated by one or more verification neural networks 410 (e.g., multiple second neural networks). For example, during verification mode, the verification network 400 compares the output generated by the neural network 405 with the output of the verification neural network 410 using the same input data. The input data may include unlabeled training data. In this example, the verification neural network 410 may be trained using training data not used to train the neural network 405.

[0085] It should be understood that neural network 405 and verification neural network 410 may comprise any suitable deep neural network 300. As shown, verification network 400 includes neural network 405, verification neural network 410, comparison module 413, and selector module 415. For example, verification network 400 may be a software program that can be loaded into memory and executed by a processor in computer 110 and / or server 145.

[0086] Selector module 415 enables verification network 400 to operate in feature mode or verification mode. In feature mode, neural network 405 receives sensor data from one or more sensors 115 via data path 420 and generates output based on the received sensor data via data path 425. For example, neural network 405 may include a CNN that receives images captured by one or more image sensors 115 via data path 420 and performs object classification based on the images. The output indicating object classification may be provided via data path 425 to one or more other software modules, and the software modules may generate control instructions for the operation of vehicle 105. For example, based on object classification, the software modules may generate control instructions that are provided to one or more actuators 120 to control the operation of vehicle 105.

[0087] In the verification mode, the selector module 415 sends a control command via the control path 430, causing the verification neural network 410 to receive sensor data via the data path 435. The selector module 415 also sends a control command via the data path 430, causing the output generated by the neural network 405 to be received by the comparison module 413 via the data path 440. Therefore, the verification neural network 410 can generate an output based on the same sensor data received by the neural network 405, i.e., the same input.

[0088] Comparison module 413 compares the output generated by verification neural network 410 with the output generated by neural network 405. Based on this comparison, comparison module 413 generates a comparison output via data path 445 indicating the difference between the output of neural network 405 and the outputs of one or more verification neural networks 410. Comparison module 413 compares the comparison output with a predetermined comparison threshold to determine whether the comparison output is greater than the predetermined comparison threshold. The predetermined comparison threshold can be selected based on empirical analysis.

[0089] If the comparison output is greater than a predetermined comparison threshold, the comparison module 413 generates an alarm and transmits the alarm and the output of the neural network 405 to the server 145. For example, the comparison module 413 may generate an alarm to indicate that the comparison output is greater than a predetermined comparison threshold for further review purposes. In various embodiments, the neural network 405 may operate in parallel with the verification neural network 410.

[0090] If the comparison output is less than or equal to a predetermined comparison threshold, the comparison module 413 transmits the comparison output to the server 145. The server 145 can initiate updates to one or more neural networks 405 based on the comparison output, such as causing the neural network 405 to update the corresponding weights and biases using a loss function that includes the comparison output.

[0091] In verification mode, neural network 405 receives unlabeled training data. For example, unlabeled training data may include sensor data 145 collected by the fleet and already uploaded to a server. In these embodiments, the baseline ground truth data for the output generated by neural network 405 is the output generated by verification neural network 410 based on the same received sensor data. Thus, during verification mode, the output of neural network 405 may not be provided to the software module used for vehicle decision-making.

[0092] As discussed above, the validation neural network 410 may include a neural network with a different architecture than the neural network 405. For example, the validation neural network 410 may be trained using a different dataset than the dataset used to train the neural network 405.

[0093] In some implementations, the selector module 415 can determine whether to operate the vehicle in a signature mode or a verification mode based on input received via data path 450. For example, server 145 can transmit control commands to selector module 415 to cause selector module 415 to switch between signature mode and verification mode. In other examples, the processor of computer 110 can send control commands to selector module 415 to cause selector module 415 to switch between signature mode and verification mode.

[0094] In various implementations, the verification network 400 can be deployed as a microservice. The computer 110 can store the verification neural network 410 in memory and load the verification neural network 410 when invoked by the selector module 415.

[0095] Figure 5 This is a flowchart of an example process 500 for verifying the output of neural network 405 during verification mode. The blocks of process 500 can be executed by computer 110. Process 500 begins at block 505, where it is determined whether verification mode has been enabled. For example, verification mode is enabled based on input received by selector module 415. This input can be provided by server 145 or another ECU.

[0096] If the verification mode is not enabled, then in box 510, the neural network 405 is loaded to operate in feature mode. In feature mode, the neural network 405 can generate an output based on sensor data. This output can be used by one or more software modules to at least partially operate the vehicle 105, i.e., control steering, acceleration, braking, etc.

[0097] In box 515, computer 110 initiates one or more communication protocols for characteristic mode operation. For example, computer 110 may initiate one or more gateway modules for interoperability purposes. The gateway modules may allow data to flow between various communication networks within vehicle 105, such as sensor gateways and / or actuator gateways.

[0098] In block 520, computer 110 operates neural network 405 in a feature mode. For example, neural network 405 receives sensor data from sensor 115 and generates an output based on the sensor data. As discussed above, in one embodiment, neural network 405 can be trained to perform object classification, and neural network 405 outputs object classification data based on sensor inputs. Using the object classification data, one or more software modules employed by computer 110 can assist in vehicle operation. In block 525, vehicle 105 is operated based on the output from neural network 405. For example, one or more software modules can generate control commands that are sent to actuator 120 to operate one or more components 125 of vehicle 105 based on the output of neural network 405. Process 500 then transitions back to block 505.

[0099] If the verification mode is enabled, then in box 530, one or more vehicle 105 actuators 120 are disengaged from neural network 405. For example, if selector module 415 receives input selecting the verification mode, software modules and / or corresponding gateway modules can be disabled to prevent the output from neural network 405 from operating vehicle 105.

[0100] In box 535, computer 110 loads the verification neural network 410. For example, for verification purposes, computer 110 may access and load the verification neural network 410 into memory. In box 540, computer 110 reconfigures the sensor data provided to one or more neural networks 405, 410. For example, depending on the type of unlabeled sensor data received for verification purposes, it may be necessary to modify the configuration of one or more neural networks. Computer 110 may modify the neural network configuration based on a configuration file provided by server 145 and / or a configuration file stored in memory.

[0101] In box 545, computer 110 causes verification network 400 to compare the output generated by neural network 405 with the output generated by one or more verification neural networks 410. It should be understood that multiple verification neural networks 410 may be used, wherein the output of neural network 405 is compared with a corresponding output from each verification neural network 410. In box 550, comparison module 413 compares the output from neural network 405 with the output from verification neural network 410. In box 555, comparison module 413 determines whether the comparison output is greater than a predetermined comparison threshold. If the comparison output is greater than the predetermined comparison threshold, in box 560, comparison module 413 generates an alarm and transmits the comparison data to server 145. Process 500 then returns to box 505. If the comparison output is not greater than the predetermined comparison threshold, comparison module 413 transmits the comparison output to server 145. Process 500 then returns to box 505.

[0102] The description in this disclosure is merely exemplary in nature, and variations thereof without departing from the spirit and scope of this disclosure are intended to be within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.

[0103] Generally, the described computing system and / or device may employ any of a variety of computer operating systems, including, but not limited to, Microsoft Automotive® operating system, the Microsoft Windows® operating system, Unix operating systems (e.g., Solaris® operating system released by Oracle Corporation of Redwood Coast, California), AIX UNIX operating system released by International Business Machines Corporation of Armonk, New York, Linux operating system, Mac OSX and iOS operating systems released by Apple Inc. of Cupertino, California, BlackBerry OS released by BlackBerry Ltd. of Waterloo, Canada, and Android operating system developed by Google and the Open Handset Alliance, or the QNX® automotive platform for infotainment provided by QNX Software Systems, Inc. Examples of computing devices include, but are not limited to, in-vehicle computers, computer workstations, servers, desktop computers, laptops, notebook computers, or handheld computers, or some other computing systems and / or devices.

[0104] Computers and computing devices typically include computer-executable instructions, which are executable by one or more computing devices, such as those listed above. Computer-executable instructions can be compiled or interpreted from computer programs created using various programming languages ​​and / or technologies, including but not limited to Java, alone or in combination. TMC, C++, Matlab, Simulink, Stateflow, Visual Basic, JavaScript, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines, such as the Java Virtual Machine and the Dalvik Virtual Machine. Typically, a processor (e.g., a microprocessor) receives instructions from memory, computer-readable media, etc., and executes those instructions to perform one or more processes, including one or more processes described herein. Such instructions and other data can be stored and transferred using a wide variety of computer-readable media. Files in a computing device are typically collections of data stored on computer-readable media such as storage media and random access memory.

[0105] Memory can include computer-readable media (also known as processor-readable media), which includes any non-transitory (e.g., tangible) medium involved in providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. Non-volatile media can include, for example, optical discs or magnetic disks and other permanent storage devices. Volatile media can include, for example, dynamic random access memory (DRAM), which typically constitutes main memory. Such instructions can be transmitted via one or more transmission media, including coaxial cables, copper wires, and optical fibers, including wires containing a system bus coupled to the processor of the ECU. Common forms of computer-readable media include, for example, floppy disks, floppy disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROM, EPROM, flash EEPROM, any other memory chip or cassette tape, or any other medium from which a computer can read.

[0106] The databases, data repositories, or other data stores described herein can include various mechanisms for storing, accessing, and retrieving a wide variety of data, including hierarchical databases, file sets in file systems, application databases in proprietary formats, relational database management systems (RDBMS), and so on. Each such data store is typically contained within a computing device employing a computer operating system such as one of those described above, and is accessed via a network in any one or more of various ways. File systems can be accessible from the computer operating system and can include files stored in various formats. In addition to the languages ​​used to create, store, edit, and execute the stored procedures, RDBMS typically employs a structured query language (SQL), such as the PL / SQL language mentioned above.

[0107] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) and stored on an associated computer-readable medium (e.g., disks, storage, etc.). A computer program product may include such instructions stored on a computer-readable medium for performing the functions described herein.

[0108] In this application, including the following definitions, the term "module" or "controller" may be replaced by the term "circuit". The term "module" may refer to, or be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores code executed by the processor circuitry; other suitable hardware components that provide the aforementioned functionality; or combinations of some or all of the foregoing, such as in a system-on-a-chip.

[0109] This module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module in this disclosure may be distributed among multiple modules connected via the interface circuits. For example, multiple modules may allow for load balancing. In another example, a server (also known as a remote or cloud) module may perform some functions on behalf of a client module.

[0110] Regarding the media, processes, systems, methods, heuristics, etc., described herein, it should be understood that although the steps of such processes are described as occurring according to an ordered sequence, such processes can be practiced using steps performed in a different order than those described herein. It should be further understood that certain steps can be performed simultaneously, i.e., other steps can be added, or certain steps described herein can be omitted. In other words, the description of the processes herein is provided for the purpose of illustrating certain embodiments and should not be construed as limiting the claims in any way.

[0111] Therefore, it should be understood that the above description is illustrative and not restrictive. Many embodiments and applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. The scope of the invention should not be determined by reference to the foregoing description, but rather by reference instead to the appended claims and the full scope of their equivalents. Future developments are anticipated and intended to occur in the field discussed herein, and the disclosed systems and methods will be incorporated into such future embodiments. In summary, it should be understood that the invention is capable of modifications and variations and is defined only by the following claims.

[0112] All terms used in the claims are intended to give the simple and common meaning as understood by one of those skilled in the art, unless expressly indicated otherwise herein. In particular, the use of singular articles such as “a,” “the,” “the,” etc., should be understood to refer to one or more of the indicated elements, unless the claims expressly limit this to the contrary.

Claims

1. A system for a vehicle, comprising a computer, the computer including a processor and a memory, the memory including instructions that program the processor to: Unlabeled sensor data is received at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data; The unlabeled sensor data is received at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode, and the second neural network is different from the first neural network; Receive the selection to enter verification mode; When entering verification mode, at least one vehicle actuator is disengaged from the first neural network to prevent the first neural network from operating the vehicle. During the verification mode, the output generated by the first neural network is compared with the output generated by the second neural network; An alarm is generated when the difference between the output generated by the first neural network and the output generated by the second neural network exceeds a predetermined comparison threshold. When the difference exceeds a predetermined comparison threshold, the first neural network is updated based on the second neural network; Selecting the input feature mode; When entering the feature mode, the at least one vehicle actuator is engaged with the first neural network, allowing the first neural network to operate the vehicle; as well as During the feature pattern, the at least one vehicle actuator is operated based on the output generated by the first neural network.

2. The system of claim 1, wherein the selection is transmitted from the server.

3. The system of claim 1, wherein the selection is transmitted from the vehicle's electronic controller unit.

4. The system of claim 1, wherein the first neural network is trained using a first dataset, and the second neural network is trained using a second dataset, wherein the second dataset is different from the first dataset.

5. The system of claim 1, wherein the unlabeled sensor data includes sensor data collected by the fleet.

6. A vehicle comprising the system according to claim 1.

7. The vehicle of claim 6, wherein the selection is transmitted from the server.

8. The system of claim 6, wherein the selection is transmitted from the vehicle's electronic controller unit.

9. The system of claim 6, wherein the first neural network is trained using a first dataset, and the second neural network is trained using a second dataset, wherein the second dataset is different from the first dataset.

10. The system of claim 6, wherein the unlabeled sensor data includes sensor data collected by the fleet.

11. A method of operating a vehicle, comprising: Unlabeled sensor data is received at a first neural network, wherein the first neural network generates an output based on the unlabeled sensor data; The unlabeled sensor data is received at a second neural network, wherein the second neural network generates an output based on the unlabeled sensor data during a verification mode, and the second neural network is different from the first neural network; Receive the selection to enter verification mode; When entering verification mode, at least one vehicle actuator is disengaged from the first neural network to prevent the first neural network from operating the vehicle. During the verification mode, the output generated by the first neural network is compared with the output generated by the second neural network; An alarm is generated when the difference between the output generated by the first neural network and the output generated by the second neural network exceeds a predetermined comparison threshold. When the difference exceeds a predetermined comparison threshold, the first neural network is updated based on the second neural network; Selecting the input feature mode; When entering the feature mode, the at least one vehicle actuator is engaged with the first neural network, allowing the first neural network to operate the vehicle; as well as During the feature pattern, the at least one vehicle actuator is operated based on the output generated by the first neural network.

12. The method of claim 11, wherein the selection is transmitted from the server.

Images