Vehicle control device and method for controlling vehicle

Abstract

The present disclosure relates to vehicle control based on a neural network. In particular, the present disclosure relates to determining confidence in the output of a neural network by comparing a firing pattern observed during operation of a vehicle with a reference firing pattern obtained by observing firing of neurons during a training phase.

Description

Vehicle control device and method for controlling a vehicle

[0001] This disclosure relates to vehicle control based on neural networks. In particular, this disclosure relates to determining the reliability of the output of a neural network by comparing firing patterns observed during vehicle operation with reference firing patterns obtained by observing neuronal firing during the training phase.

[0002] Japanese Patent Publication No. 2022-014974 describes an evaluation device comprising a first acquisition unit for acquiring a first index, a second acquisition unit for acquiring a second index, and an evaluation unit for evaluating reliability. The first index represents the difference in data space between training input data and actual operation input data. The second index represents the difference in the firing tendency of neurons in a neural network learning model between when training input data is input and when actual operation input data is input. The evaluation unit evaluates the reliability of the predicted values ​​output from the learning model for the actual operation input data based on the first index acquired by the first acquisition unit and the second index acquired by the second acquisition unit.

[0003] In the background technology described above, when data deviating from the training input data is supplied to the neural network, it is impossible to determine the deviation in the estimation results caused by the firing of neurons that were not used during the training phase. This problem should be considered when applied to vehicle control systems that require extremely high safety standards.

[0004] Aspects of this disclosure relate to vehicle control devices, methods for controlling vehicles, and computer-readable media.

[0005] According to a first aspect of this disclosure, the vehicle control device may include a control data generation unit provided to generate control data by inputting data sensed by a sensor into a neural network having a plurality of intermediate layers; a controller provided to control the vehicle based on the control data; a confidence calculation unit provided to calculate the confidence level of the output of the neural network by comparing a first firing pattern in a first half-layer of one of the intermediate layers during the control phase with a first reference firing pattern stored in the confidence calculation unit, and comparing a second firing pattern in a second half-layer of one of the intermediate layers during the control phase with a second reference firing pattern stored in the confidence calculation unit, wherein the first reference firing pattern is generated based on a firing pattern in a first half-layer of one of the intermediate layers during the training phase, and the second reference firing pattern is generated based on a firing pattern in a second half-layer of one of the intermediate layers during the training phase; and a control data determination unit provided to determine whether or not the controller should use the control data based on the confidence level.

[0006] In this regard, the term “vehicle control device” as used throughout this specification and the claims may, in particular, refer to a device that controls the operation of a vehicle, at least partially. For example, a vehicle control device may generate or output signals that cause the state of actuators within the vehicle to change, thereby, for example, accelerating or decelerating the vehicle, or changing the direction of travel. Alternatively or additionally, a vehicle control device may generate or output signals that cause a change in the process by which energy is converted from one form to another (e.g., chemical to electrical, or electrical to mechanical). For example, a vehicle control device may output signals that change the operation of an internal combustion engine, fuel cell, or battery (network). Furthermore, the term “sensor” as used throughout this specification and the claims may, in particular, refer to a circuit configured to monitor a part of the vehicle or a part of the vehicle's environment and provide data representing them. For example, a sensor may be an external sensor (e.g., a camera) that monitors the area around the vehicle in relation to other vehicles in an attempt to avoid a collision, or a sensor may be an internal sensor that measures the temperature of the vehicle's internal combustion engine, fuel cell, or battery (network), the vehicle's speed, the torque of the vehicle's motor, etc.

[0007] Furthermore, the term “control data generation unit” as used throughout this specification and the claims may, in particular, refer to hardware, or a combination of hardware and software, configured to generate or output control data. In this regard, the term “control data” as used throughout this specification and the claims refers to data that is particularly intended to control the manner in which a vehicle operates. For example, the control data may be provided to a controller that processes the data to generate or output signals, for example, signals that cause a change in the state of an actuator in a vehicle, or signals that cause a change in the process of converting energy from one form to another. In this regard, the term “controller” as used throughout this specification and the claims refers in particular to a circuit including a processor, memory, and input / output peripherals. Output peripherals may be connected (directly or indirectly) to actuators or energy conversion devices controlled by the controller.

[0008] Furthermore, the term “confidence calculation unit” as used throughout this specification and the claims may, in particular, refer to hardware, or a combination of hardware and software, configured to calculate or output a confidence level assigned to control data. The confidence level may be expressed as a numerical value within a predetermined range or from a predetermined set of values. The confidence level may be evaluated in conjunction with a risk level inherent to unreliable output data. For example, even slightly unreliable output data may be discarded if, otherwise, the output data would cause the vehicle or vehicle component to be associated with a high risk. Furthermore, the term “control data determination unit” as used throughout this specification and the claims may, in particular, refer to hardware, or a combination of hardware and software, configured to generate or output data indicating whether control data can be used. Note that some or all of the control data generation unit, confidence calculation unit, and control data determination unit may be implemented by software running on a single computer. Furthermore, the single computer may host a neural network.

[0009] In this regard, the term “neural network” as used throughout this specification and the claims may, in particular, refer to a network of interconnected nodes (neurons), whether physical or virtual, where each neuron is configured to process signals received from preceding firing neurons (connected to the neuron) and output those signals to subsequent neurons. The neural network may be a trained neural network adapted to its task during a training phase based on pre-collected or artificially generated training data, including input data and (desired) output data (target data). The neural network may be part of a trained artificial intelligence (AI) unit or part of a trained machine learning (ML) unit. The vehicle control device may have two (redundant) neural networks and switch between them when the confidence level of the output data of one of the networks falls below a predetermined lower limit.

[0010] A neural network may be divided into layers where neurons in subsequent layers receive signals from firing neurons in preceding layers. For example, signals may travel from an input layer through two or more (hidden) hidden layers to an output layer. A hidden layer may be divided into two half-layers. A first half-layer may comprise all hidden layers closer to the input layer than the output layer (with respect to the hidden layers between the input and output layers). A second half-layer may comprise all hidden layers closer to the output layer than the input layer (with respect to the hidden layers between the input and output layers). In the case of a non-uniform number of hidden layers, the middle hidden layer may be ignored, divided between half-layers, or joined to either the first or second half-layer for the purposes of this disclosure.

[0011] Furthermore, the term “firing pattern” as used throughout this specification and the claims may, in particular, refer to a representation of firing neurons as opposed to non-firing neurons, and vice versa. For example, a firing pattern may be provided in the form of a list or vector that includes / indicates firing neurons and / or non-firing neurons. Instead of a digital representation in which neurons are listed as either firing or not firing, a reference firing pattern may include the probability of each neuron in a layer firing or not firing.

[0012] Furthermore, the term “training stage” as used throughout this specification and the claims may, in particular, refer to a pre-operation stage in which a neural network can be adapted to its task based on pre-collected or artificially generated data (and without outputting control data to a controller), and in which other preparations for the use of the neural network, such as the generation of a reference firing pattern, can be performed.

[0013] By splitting the output along the intermediate layers of the neural network, the confidence level of the neural network's estimation results can be calculated more accurately and in real time, enabling instantaneous responses in vehicle control.

[0014] One of the hidden layers may have a first half-layer that follows the input layer of the neural network, and the output layer of the neural network may have a second half-layer that follows one of the hidden layers. In particular, one of the hidden layers may have a first half-layer that follows immediately after the input layer of the neural network, and the output layer of the neural network may have a second half-layer that follows immediately after one of the hidden layers.

[0015] The reliability calculation unit may output a first evaluation index calculated based on the comparison result between the first firing pattern and the first reference firing pattern, and a second evaluation index calculated based on the comparison result between the second firing pattern and the second reference firing pattern.

[0016] The confidence calculation unit may estimate the confidence level by selecting the higher of the first and second evaluation indices, or the lower of the first and second evaluation indices.

[0017] The first evaluation index may be calculated using the Euclidean distance between the first reference ignition pattern and the first ignition pattern, and the second evaluation index may be calculated using the Euclidean distance between the second reference ignition pattern and the second ignition pattern.

[0018] The control data determination unit may include a control switching unit that outputs a signal for switching control based on the confidence level and first and second evaluation indices.

[0019] The first reference ignition pattern may be selected from a first group of reference ignition patterns based on its similarity to the first ignition pattern, and the second reference ignition pattern may be selected from a second group of reference ignition patterns based on its similarity to the second ignition pattern.

[0020] The vehicle may be an automobile.

[0021] According to a second aspect of the present disclosure, a vehicle control device may include a sensor that outputs sensor data, a neural network having an input layer, an output layer, and an intermediate layer, and a controller that controls the vehicle, wherein sensor data or data derived from sensor data is input to the input layer, the neural network generates control data based on the data input to the input layer, the output layer outputs the control data, and the control data is not trusted by the controller if it does not pass a similarity test which includes comparing a first firing pattern of a first intermediate layer among the intermediate layers with a first reference firing pattern, and comparing a second firing pattern of a second intermediate layer among the intermediate layers with a second reference firing pattern.

[0022] The first hidden layer may be closer to the input layer than the output layer, and the second hidden layer may be closer to the output layer than the input layer.

[0023] The first hidden layer may follow immediately after the input layer, and the output layer may follow immediately after the second hidden layer.

[0024] A first evaluation index may be calculated based on a comparison between a first firing pattern of a first intermediate layer and a first reference firing pattern, and a second evaluation index may be calculated based on a comparison between a second firing pattern of a second intermediate layer and a second reference firing pattern, and an alternative control strategy that can be adopted by the controller when the control data is not trusted by the controller may be selected, taking into consideration the first and second evaluation indices.

[0025] The first reference ignition pattern may be selected from a first group of reference ignition patterns based on its similarity to the first ignition pattern, and the second reference ignition pattern may be selected from a second group of reference ignition patterns based on its similarity to the second ignition pattern.

[0026] The vehicle may be an automobile.

[0027] According to a third aspect of the present disclosure, a method for controlling a vehicle may include the steps of: inputting first data into the input layer of a neural network; performing a first comparison between a first firing pattern of a first hidden layer of the neural network and a first reference firing pattern obtained by observing the firing of neurons in the first hidden layer during a training phase; performing a second comparison between a second firing pattern of a second hidden layer of the neural network and a second reference firing pattern obtained by observing the firing of neurons in the second hidden layer during a training phase; and determining whether second data generated by the neural network based on the first data should be discarded based on the first and second comparisons.

[0028] The first hidden layer may be closer to the input layer than the output layer, and the second hidden layer may be closer to the output layer than the input layer.

[0029] The first intermediate layer among the intermediate layers may follow immediately after the input layer, and the output layer may follow immediately after the second intermediate layer among the intermediate layers.

[0030] The method may further include a step of calculating a first evaluation index based on a first comparison, and a step of calculating a first evaluation index based on a first comparison.

[0031] The first evaluation index may be calculated by the Euclidean distance between the first reference firing pattern and the first firing pattern, and the second evaluation index may be calculated by the Euclidean distance between the second reference firing pattern and the second firing pattern.

[0032] The method may further include a step of calculating a reliability level of the output of the neural network based on the first evaluation index and the second evaluation index, and a step of comparing the reliability level with a threshold value.

[0033] The first reference firing pattern may be based on the averaging over the firing of the neurons in the first intermediate layer during the training phase, and the second reference firing pattern may be based on the averaging over the firing of the neurons in the second intermediate layer during the training phase.

[0034] The first reference firing pattern may be selected from a first group of reference firing patterns based on the similarity to the first firing pattern, and the second reference firing pattern may be selected from a second group of reference firing patterns based on the similarity to the second firing pattern.

[0035] The vehicle may be an automobile.

[0036] According to a fourth aspect of the present disclosure, a computer-readable medium may store instructions that cause a computer to perform a method for controlling a vehicle when executed by the computer.

[0037] In particular, the features described throughout this specification with respect to the method may be features of a vehicle control device, and vice versa.

[0038] Many of the aforementioned embodiments and their associated advantages will be better understood and more readily apparent by referring to the following description of embodiments, which will be interpreted in conjunction with the accompanying drawings, and in the accompanying drawings, unless otherwise specified, similar reference numbers refer to the same parts throughout the various figures. An example of a similarity test that may be used to determine the reliability of data generated by a neural network is illustrated. An example of the calculation of a reliability metric that may be used in a similarity test is illustrated. An example of the application of a reliability metric is illustrated. An example of countermeasures that may be triggered by a specific reliability value obtained by applying a reliability metric is illustrated. A block diagram of a vehicle control device that uses a similarity test to determine the reliability of an output based on a neural network is shown. A flowchart of a method for determining a reference pattern is shown. A flowchart of a process for controlling a vehicle is shown. A flowchart of another process for controlling a vehicle is shown.

[0039] In particular, not all drawings are drawn to scale, and unless otherwise indicated, they are intended only to conceptually illustrate the structures and procedures described herein.

[0040] Figure 1 shows a neural network 10 undergoing training. During the training phase, training data (10,000+ points) is input to the neural network 10, and the neural network 10 is updated with the aim of training it to accurately predict target data (10,000+ points). Furthermore, the training phase is also used to generate baseline firing patterns 12 and 14.

[0041] Incidentally, the reference firing patterns 12 and 14 may be generated after the update of the neural network 10 is completed. The reference firing pattern 12 covers the firing of one or more intermediate layer neurons closer to the input layer than the output layer, averaged over one or more data points. The reference firing pattern 14 covers the firing of one or more intermediate layer neurons closer to the output layer than the input layer, averaged over one or more data points. Therefore, the reference firing patterns 12 and 14 indicate neuron activities triggered by one or more data points in the first half layer and the second half layer of the intermediate layer. Here, the function of the input layer can process input data from external sensors and internal sensors so as to conform to a data structure suitable for the neural network. Also, the function of the output layer can reconstruct the result into control data that can be processed by the controller.

[0042] For example, when the output of the neural network 10 is used to control a vehicle, etc., at the time when the neural network 10 is operating, by using the reference firing patterns 12 and 14, the similarity between the firing pattern 16 shown by one or more intermediate layers closer to the input layer than the output layer and the reference firing pattern 12, and the firing pattern 18 shown by one or more intermediate layers closer to the output layer than the input layer and the reference firing pattern 14 are evaluated, the reliability of the neural network 10 may be determined. Both similarity evaluations may be aggregated into a reliability value that can be used to determine whether to use, correct, or discard the output of the neural network 10.

[0043] As shown in FIG. 2, the reference firing patterns 12, 14 and the firing patterns 16, 18 are vectors e １ , e ２ , a １ , and a ２ , and may be represented by, the evaluation indices R １ and R ２ may be calculated as in equations (1) and (2), d １ is the number of neurons in the first half layer, and d ２ is the number of neurons in the second half layer.

[0044]

[0045] Next, the overall reliability value may be calculated as R = R １ R ２ . As shown in FIG. 3, the reliability value R may be used to determine whether and how to use the output of the neural network 10. This determination may be made based on a comparison between the reliability value R and a predetermined value. If the output of the neural network 10 cannot be used as it is due to its low reliability level (the reliability value R is lower than the predetermined value), the previous output may be used instead, or a correction factor may be applied to the output. Further, the evaluation indices R １ and R ２ may be used to determine whether the cause of the low reliability level lies in the input data or in the neural network 10.

[0046] As shown in FIG. 4, when the first evaluation index is low and the second evaluation index is high (suggesting that there is an abnormally large deviation between the firing pattern 16 shown by one or more intermediate layers closer to the input layer than the output layer and the reference firing pattern 12), this may suggest that the input data is outside the range of the training data. In this case, the output may be ignored, the neural network 10 may be retrained, the sensor may be checked for errors, or the training data may be re-evaluated. Conversely, when the first evaluation index is high and the second evaluation index is low (suggesting that there is an abnormally large deviation between the firing pattern 18 shown by one or more intermediate layers closer to the output layer than the input layer and the reference firing pattern 14), this may suggest that there is a problem with the specifications of the neural network 10 (such as overfitting, underlearning, small architecture size, etc.). In this case, the output may be ignored and the neural network 10 may be retrained. Both evaluation indices R １ and R ２If the value is low, and therefore indicates that the input and output data are outside the range of the training data, the model may need to be modified, control may need to be stopped, the sensors may need to be checked for defects, and / or the training data may need to be corrected.

[0047] Figure 5 shows a block diagram of the vehicle control device 20. The vehicle control device 20 includes a control data generation unit 22 that generates control data from data provided by one or more external sensors 24 and one or more internal sensors 26. The data sensed by the sensors 24 and 26 is input to a neural network 10, and the neural network 10 outputs the data to a control data determination unit 28. The control data determination unit 28 determines whether the controller 32 should use the output of the neural network 10 based on the confidence level received from the confidence calculation unit 30. If the controller 32 uses the output of the neural network 10, the controller 32 may use it to control, for example, an internal combustion engine 34 or one or more actuators 36. This invention may also be applied to image processing for autonomous vehicles.

[0048] Figure 6 shows a flowchart of the method for determining the baseline pattern. The method begins with predicting the training data in step 38, followed by step 40, which updates the neuron weight coefficients. Both steps 38 and 40 are repeated until training is complete. Once training is complete, it is determined whether the firing patterns should be clustered. If not, the baseline firing pattern is calculated in step 42 by averaging across the firing patterns. If the firing patterns are clustered, the firing patterns are stored in groups using the K-means method in step 44. Then, in step 46, the baseline firing pattern is calculated by averaging across the firing patterns in each group.

[0049] Figure 7 shows a flowchart of the process for controlling the vehicle. The process begins in step 48 with loading reference firing patterns 12 and 14. In step 50, input data is supplied to the neural network 10, and the neural network 10 provides output data. In step 52, the firing patterns 16 and 18 of the hidden layer are observed, and in step 54, the observed firing patterns 16 and 18 are compared with the reference firing patterns 12 and 14. If clustering is performed as described with reference to steps 44 and 46 of Figure 6, the observed firing patterns 16 and 18 are compared with the reference firing patterns 12 and 14 that have the highest degree of similarity. In step 54, a confidence level is determined, and in step 56, it is determined whether or not to use the output of the neural network 10 based on the confidence level.

[0050] Figure 8 shows a flowchart of another process for controlling a vehicle. The process begins in step 58 with inputting data into the input layer of the neural network 10. In steps 60 and 62, the observed firing patterns 16 and 18 are compared with reference firing patterns 12 and 14. If clustering is performed as described with reference to steps 44 and 46 of Figure 6, the observed firing patterns 16 and 18 are compared with the reference firing patterns 12 and 14 that have the highest degree of similarity. In step 70, based on the comparison, it is determined whether the output of the neural network 10 should be discarded.

[0051] Furthermore, it should be noted that the examples in this disclosure may take the form of entirely hardware examples, entirely software examples (including firmware, resident software, microcode, etc.), or examples combining software and hardware aspects. Moreover, the examples in this disclosure may take the form of computer program products on computer-readable media having computer executable program code embodied on the medium.

[0052] It should be noted that arrows may be used in drawings to represent communications, transfers, or other activities involving two or more entities. Double-headed arrows generally indicate that an activity can occur in both directions (e.g., a command / request in one direction with a corresponding response returned in the other, or peer-to-peer communication initiated by either entity), although in some situations the activity may not necessarily occur in both directions.

[0053] While single-headed arrows generally indicate one-way activity, it should be noted that in certain situations, such directional activity may actually include bidirectional activity (e.g., a message from sender to receiver and an acknowledgment from receiver to sender, or the establishment of a connection before forwarding and the termination of a connection after forwarding). Therefore, the type of arrow used in a particular drawing to represent a specific activity is illustrative and should not be considered limiting.

[0054] Embodiments / examples have been described above with reference to flowcharts and / or block diagrams of methods and apparatus. It can be understood that each block in the flowcharts and / or block diagrams, and / or combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer executable program code.

[0055] Computer executable program code may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device for manufacturing a particular machine, and as a result, the program code executed through the processor of the computer or other programmable data processing device creates means for performing functions / operations / outputs specified in a flowchart, block diagram block or a set of blocks, diagrams, and / or a written description.

[0056] These computer-executable program codes may also be stored in computer-readable memory that can instruct a computer or other programmable data processing device to function in a particular way, and as a result, program codes stored in computer-readable memory can be used to manufacture products that include instruction means to perform functions / operations / outputs specified in flowcharts, block diagram blocks, figures, and / or descriptions.

[0057] Computer-executable program code can also be loaded into a computer or other programmable data processing device to generate a computer implementation process of a set of operational steps to be executed on the computer or other programmable device, so that the program code executed on the computer or other programmable device provides steps for implementing functions / operations / outputs specified in the flowchart, block diagram blocks, figures, and / or specification. Alternatively, the computer program implementation steps or operations may be combined with steps or operations performed by an operator or human to carry out the embodiment.

[0058] Communication networks may generally include public networks and / or private networks, and may include local area, wide area, metropolitan area, storage, and / or other types of networks, and may use communication technologies including, but not limited to, analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0059] It should also be noted that the device may use communication protocols and messages (e.g., messages created, transmitted, received, stored, and / or processed by the device), and such messages may be transmitted by communication networks or media.

[0060] Unless the context requires otherwise, this disclosure should not be construed as being limited to any particular communication message type, communication message format, or communication protocol. Therefore, communication messages may, in general, include, but are not limited to, frames, packets, datagrams, user datagrams, cells, or other types of communication messages.

[0061] Unless the context requires otherwise, references to specific communication protocols are illustrative, and alternative embodiments may, if necessary, be variations of such communication protocols (e.g., modifications or extensions to protocols that may be made from time to time), or other protocols known or developed in the future.

[0062] Logical flows may be described herein to demonstrate various aspects, and it should be noted that this disclosure should not be construed as limiting it to any particular logical flow or logical implementation. The described logic may be divided into different logical blocks (e.g., programs, modules, functions, or subroutines) without altering the overall result.

[0063] In many cases, logical elements can be added, modified, omitted, executed in a different order, or implemented using different logical structures (e.g., logic gates, looping primitives, conditional logic, and other logical structures) without changing the overall result.

[0064] This disclosure is not limited to, but may be implemented in many different forms, including computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general-purpose computer), programmable logic for use with a programmable logic device (e.g., a field-programmable gate array (FPGA) or other PLD), individual components, integrated circuits (e.g., application-specific integrated circuits (ASICs)), or any other means including any combination thereof. Computer program logic that implements some or all of the described functions is typically translated into a computer-executable form, stored in a computer-readable medium, and implemented as a set of computer program instructions executed by a microprocessor under the control of an operating system. Hardware-based logic that implements some or all of the described functions may be implemented using one or more appropriately configured FPGAs.

[0065] Computer program logic that implements all or part of the functions described herein may be embodied in a variety of forms, including but not limited to source code, computer executable, and various intermediate forms (e.g., forms generated by an assembler, compiler, linker, or locator).

[0066] The source code may include a set of computer program instructions implemented in one of several programming languages ​​for use in various operating systems or operating environments (e.g., object code, assembly language, or high-level languages ​​such as Fortran, C, C++, Java, or HTML). The source code may define and use various data structures and communication messages. The source code may be in computer executable format (e.g., via an interpreter), or the source code may be converted to computer executable format (e.g., via a translator, assembler, or compiler).

[0067] Computer executable program code for performing the operations of the embodiments of this disclosure may be written in an object-oriented, scripting, or non-scripting programming language such as Java, Perl, Smalltalk, or C++. However, computer executable program code for performing the operations of the embodiments may also be written in a conventional procedural programming language such as the C programming language or a similar programming language.

[0068] Computer program logic that implements all or part of the functions described herein may run on a single processor (e.g., simultaneously) at different times, or on multiple processors at the same or different times, under a single operating system process / thread, or under different operating system processes / threads.

[0069] Therefore, the term “computer process” or similar expression can generally refer to the execution of a set of computer program instructions, regardless of whether different computer processes run on the same or different processors, and regardless of whether different computer processes run under the same operating system process / thread or different operating system processes / threads.

[0070] Computer programs may be permanently or temporarily stored in any form (e.g., source code, computer executable, or intermediate) on tangible storage media such as semiconductor memory devices (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), magnetic memory devices (e.g., diskettes or fixed disks), optical memory devices (e.g., CD-ROMs), PC cards (e.g., PCMCIA cards), or other memory devices.

[0071] Computer programs may be provided in any form of signals that can be transmitted to a computer using any of a variety of communication technologies, including but not limited to analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0072] Computer programs may be distributed in any form as removable storage media with accompanying printed or electronic documentation (e.g., shrink-wrapped software), pre-loaded onto a computer system (e.g., on a system ROM or fixed disk), or distributed from a server or electronic bulletin board via a communication system (e.g., the Internet or the World Wide Web).

[0073] Hardware logic (including programmable logic for use in programmable logic devices) that implements all or part of the functions described herein may be designed using conventional manual methods, or may be designed, captured, simulated, or documented electronically using various tools such as computer-aided design (CAD), hardware description languages ​​(e.g., VHDL or AHDL), or PLD programming languages ​​(e.g., PALASM, ABEL, or CUPL).

[0074] Any suitable computer-readable medium can be used. This computer-readable medium may, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, devices, or media.

[0075] More specific examples of computer-readable media include, but are not limited to, electrical connections having one or more wires, or other tangible storage media such as portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), compact disc read-only memory (CD-ROM), or other optical or magnetic storage devices.

[0076] Programmable logic may be permanently or temporarily provided in tangible storage media such as semiconductor memory devices (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), magnetic memory devices (e.g., diskettes or fixed disks), optical memory devices (e.g., CD-ROMs), or other memory devices.

[0077] Programmable logic may be provided in signals that can be transmitted to a computer using any of a variety of communication technologies, including but not limited to analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0078] The programmable logic may be distributed as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink-wrapped software), pre-loaded onto a computer system (e.g., on a system ROM or fixed disk), or distributed from a server or electronic bulletin board via a communication system (e.g., the Internet or the World Wide Web). Of course, some embodiments may be implemented as a combination of both software (e.g., computer program products) and hardware. Still other embodiments may be implemented entirely as hardware or entirely as software.

[0079] While certain exemplary embodiments are described and shown in the accompanying drawings, it should be understood that such embodiments are illustrative and not limited to the specific structures and arrangements shown and described, as various other changes, combinations, omissions, modifications, and substitutions are possible in addition to those described in the paragraphs above.

[0080] Those skilled in the art will understand that various adaptations, modifications, and / or combinations of the embodiments described herein can be formed. Therefore, it should be understood that within the scope of the appended claims, the disclosure may be implemented in ways other than those specifically described herein. For example, unless otherwise specified, the steps of the processes described herein may be performed in an order different from that described herein, and one or more steps may be combined, separated, or performed simultaneously. Furthermore, as those skilled in the art will see, different examples or embodiments described herein can be combined to form other examples by considering the disclosure.

[0081] 10 Neural network, 12 Reference ignition pattern, 14 Reference ignition pattern, 16 Ignition pattern, 18 Ignition pattern, 20 Vehicle control device, 22 Control data generation unit, 24 Sensor, 26 Sensor, 28 Control data judgment unit, 30 Confidence calculation unit, 32 Controller, 34 Internal combustion engine, 36 Actuator, 38 Step, 40 Step, 42 Step, 44 Step, 46 Step, 48 Step, 50 Step, 52 Step, 54 Step, 56 Step, 58 Step, 60 Step, 62 Step, 64 Step, 66 Step, 68 Step, 70 Step

Claims

1. A control data generation unit (22) is provided to generate control data by inputting data sensed by sensors (24, 26) into a neural network (10) having multiple intermediate layers; a controller (32) is provided to control the vehicle based on the control data; and a confidence calculation unit (30) is provided to calculate the confidence level of the output of the neural network (10) by comparing a first firing pattern (16) in one of the first half-layers of the intermediate layers during the control phase with a first reference firing pattern (12) stored in the confidence calculation unit (30), and by comparing a second firing pattern (18) in one of the second half-layers of the intermediate layers during the control phase with a second reference firing pattern (14) stored in the confidence calculation unit (30), wherein the first reference firing pattern (12) is generated based on the firing pattern in one of the first half-layers of the intermediate layers during the training phase. A vehicle control device (20) comprising: a confidence calculation unit (30) which generates the second reference firing pattern (14) based on the firing pattern in one of the second half layers of the intermediate layer during the training stage; and a control data determination unit (28) which is provided to determine whether or not the controller (32) should use the control data based on the confidence level.

2. The vehicle control device (20) according to claim 1, wherein one first half-layer of the intermediate layer follows the input layer of the neural network (10), and the output layer of the neural network (10) follows one second half-layer of the intermediate layer.

3. The vehicle control device (20) according to at least one of claims 1 to 2, wherein the reliability calculation unit (30) outputs a first evaluation index calculated based on the comparison result of the first ignition pattern (16) and the first reference ignition pattern (12), and a second evaluation index calculated based on the comparison result of the second ignition pattern (18) and the second reference ignition pattern (14).

4. The vehicle control device (20) according to claim 3, wherein the reliability calculation unit (30) estimates the reliability level by selecting the higher evaluation index of the first and second evaluation indices, or the lower evaluation index of the first and second evaluation indices.

5. The vehicle control device (20) according to claim 3 or 4, wherein the first evaluation index is calculated by the Euclidean distance between the first reference ignition pattern (12) and the first ignition pattern (16), and the second evaluation index is calculated by the Euclidean distance between the second reference ignition pattern (14) and the second ignition pattern (18).

6. The vehicle control device (20) according to at least one of claims 3 to 5, wherein the control data determination unit (28) has a control switching unit that outputs a signal for switching control based on the confidence level and the first and second evaluation indices.

7. A vehicle control device (20) comprising: sensors (24, 26) that output sensor data; a neural network (10) having an input layer, an output layer, and an intermediate layer; and a controller (32) that controls a vehicle, wherein the sensor data or data derived from the sensor data is input to the input layer; the neural network (10) generates control data based on the data input to the input layer; the output layer outputs the control data; and the control data is not trusted by the controller (32) if it does not pass a similarity test which includes comparing a first firing pattern (16) of a first intermediate layer among the intermediate layers with a first reference firing pattern (12) and comparing a second firing pattern (18) of a second intermediate layer among the intermediate layers with a second reference firing pattern (14).

8. The vehicle control device (20) according to claim 7, wherein the first intermediate layer of the intermediate layers is closer to the input layer than the output layer, and the second intermediate layer of the intermediate layers is closer to the output layer than the input layer.

9. The vehicle control device (20) according to claim 8, wherein the first intermediate layer of the intermediate layers follows immediately after the input layer, and the output layer follows immediately after the second intermediate layer of the intermediate layers.

10. A vehicle control device (20) according to at least one of claims 7 to 9, wherein a first evaluation index is calculated based on a comparison between a first ignition pattern (16) of the first intermediate layer of the intermediate layer and a first reference ignition pattern (12); a second evaluation index is calculated based on a comparison between a second ignition pattern (18) of the second intermediate layer of the intermediate layer and a second reference ignition pattern (14); and an alternative control strategy is selected, taking into consideration the first evaluation index and the second evaluation index, to be employed by the controller when the control data is not trusted by the controller.

11. The vehicle control device (20) according to at least one of claims 1 to 10, wherein the first reference ignition pattern (12) is selected from a first group of reference ignition patterns based on its similarity to the first ignition pattern (16), and the second reference ignition pattern (14) is selected from a second group of reference ignition patterns based on its similarity to the second ignition pattern (18).

12. A vehicle control device (20) according to at least one of claims 1 to 11, wherein the vehicle is an automobile.

13. A method for controlling a vehicle, comprising:

13. Inputting first data into the input layer of a neural network (10) (58); performing a first comparison (60) between a first firing pattern (16) of a first hidden layer of the neural network (10) and a first reference firing pattern (12) obtained by observing the firing of neurons in the first hidden layer during a training phase; 24. Performing a second comparison (62) between a second firing pattern (18) of a second hidden layer of the neural network and a second reference firing pattern (14) obtained by observing the firing of neurons in the second hidden layer during a training phase; and 35. Determining (64) whether second data generated by the neural network (10) based on the first data should be discarded based on the first and second comparisons.

14. A method for controlling a vehicle according to claim 13, wherein the first intermediate layer of the intermediate layers is closer to the input layer than the output layer, and the second intermediate layer of the intermediate layers is closer to the output layer than the input layer.

15. A method for controlling a vehicle according to claim 14, wherein the first intermediate layer of the intermediate layers follows immediately after the input layer, and the output layer follows immediately after the second intermediate layer of the intermediate layers.

16. A method for controlling a vehicle according to any one of claims 13 to 15, further comprising the steps of: calculating a first evaluation index based on the first comparison; and calculating a first evaluation index based on the first comparison.

17. A method for controlling a vehicle according to claim 16, wherein the first evaluation index is calculated by the Euclidean distance between the first reference ignition pattern (12) and the first ignition pattern (16), and the second evaluation index is calculated by the Euclidean distance between the second reference ignition pattern (14) and the second ignition pattern (18).

18. A method for controlling a vehicle according to claim 17, further comprising the steps of: calculating a confidence level of the output of the neural network based on the first evaluation index and the second evaluation index; and comparing the confidence level with a threshold.

19. A method for controlling a vehicle according to any one of claims 13 to 18, wherein the first reference firing pattern (12) is based on averaging over firings of neurons in the first intermediate layer during the training phase, and the second reference firing pattern (14) is based on averaging over firings of neurons in the second intermediate layer during the training phase.

20. A method for controlling a vehicle according to any one of claims 13 to 19, wherein the first reference ignition pattern (12) is selected from a first group of reference ignition patterns based on its similarity to the first ignition pattern (16), and the second reference ignition pattern (14) is selected from a second group of reference ignition patterns based on its similarity to the second ignition pattern (18).

21. A method for controlling a vehicle according to any one of claims 13 to 20, wherein the vehicle is an automobile.

22. A computer program product or a computer-readable medium for storing the computer program product, which causes the computer to perform the method described in any one of claims 13 to 21 when executed by the computer.

Images