Systems and methods for object detection in autonomous vehicles

Abstract

Embodiments for reducing unauthorized modification of an object detection system for a motor vehicle are disclosed. As an example, a method includes: generating a cryptographic signature for an output of a node of a fully connected layer of a neural network of a vehicle object detection system, the cryptographic signature being based in part on a first private key stored in a replay protected memory block (RPMB), the output at least partially describing a detected object; and in response to verifying the cryptographic signature, adjusting vehicle operation based on the detected object. In this way, the output of the neural network can be verified such that unauthorized modifications are detected and system accuracy is improved.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to Indian Patent Application No. 202041035296, filed on August 17, 2020, entitled "Reliable Object Detection for Autonomous Driving". The entire content of the application listed above is incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure relates to the field of autonomous (e.g., self - driving) vehicles. Background Art

[0004] Some motor vehicles may include systems for environmental perception and object detection. For example, a vehicle may include an object detection system for detecting objects around the vehicle, such as for adjusting vehicle operation. Specifically, the object detection system may include a neural network for identifying and classifying objects in camera images. For example, a vehicle may include a camera and other sensors, such as lidar, radar, etc., and a neural network for processing camera images to detect objects. Driver assistance systems (e.g., such as collision avoidance systems, lane - change warning systems, cruise control systems, etc.) and autonomous driving systems may use the results of the object detection system to adjust vehicle operation. As an example, a planned vehicle trajectory may be adjusted based on the objects detected by the object detection system. Additionally, the type of adjustment may depend on the type of object detection. For example, the vehicle may be adjusted differently in response to detecting a pedestrian on the road as compared to detecting a vehicle on the road. However, in some examples, the results of the object detection system may be modified by an unauthorized entity. For example, an unauthorized modification may adjust the results of the object detection system, and vehicle controls may be adjusted based on the unauthorized modification. As a result, the accuracy of the object detection system is reduced, and customer satisfaction with vehicle operation is reduced. Additionally, the safety of autonomous vehicles is compromised. Summary of the Invention

[0005] An embodiment of a method is disclosed that includes generating a cryptographic signature for an output of a node of a fully connected layer of a neural network of a vehicle object detection system, the cryptographic signature being based in part on a first private key stored in a replay protected memory block (RPMB), the output at least partially describing a detected object; and adjusting vehicle operation based on the detected object in response to verifying the cryptographic signature. In this way, unauthorized modifications to the object detection system can be reduced. For example, by using an encryption mechanism to cryptographically sign the output of a node of a fully connected layer of a neural network of an object detection system. Additionally, by verifying the cryptographically signed output of an object detection system, unauthorized modifications can be detected. For example, an unauthorized entity cannot generate a valid cryptographic signature. Thus, it can be determined that a modification has been made to the output of an object detection system with an unverified cryptographic signature. By reducing the incidence of unauthorized modifications to the output of an object detection system, vehicle performance (especially in terms of safety) can be improved and overall customer satisfaction can be increased.

[0006] In another embodiment, a method includes: inputting a camera image at a vehicle into a neural network of an object detection system of the vehicle, the neural network including a fully connected layer; signing a first output of a node of the fully connected layer with a first cryptographic signature; signing a second output of a node of the fully connected layer with a second cryptographic signature; processing each of the first output and the second output via an activation function, an output of the activation function including a localization output signed with the first cryptographic signature and a classification output signed with the second cryptographic signature, each of the localization output and the classification output at least partially describing a detected object; verifying each of the first cryptographic signature and the second cryptographic signature based on at least one public key and at least one private key; and adjusting at least one vehicle control based on each of the localization output and the classification output in response to successfully verifying each of the first cryptographic signature and the second cryptographic signature.

[0007] In another embodiment, a system includes: a vehicle system; a vehicle control system including at least one of an autonomous vehicle control system and a driver assistance system; a plurality of sensors communicatively coupled to the vehicle control system, the plurality of sensors including at least one camera; an object detection system including a neural network that takes a camera image from at least one camera as an input and generates as outputs output coordinates corresponding to a bounding box of a detected object and its object classification; a controller storing executable instructions in a non-transitory memory that, when executed, cause the controller: adjust at least one of the autonomous vehicle control system and the driver assistance system based on the output in response to verifying a cryptographic signature of the neural network output, and generate a cryptographic signature at a fully connected layer of the neural network.

[0008] The above advantages and other advantages and features of this description will be readily apparent from the following specific embodiments, taken alone or in conjunction with the accompanying drawings. It should be understood that the above Summary is provided to introduce in a simplified form a series of concepts that are further described in the specific embodiments. It is not meant to identify the key or essential features of the claimed subject matter, the scope of which is defined by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present disclosure may be better understood by reference to the following description of non-limiting embodiments read in conjunction with the accompanying drawings, in which:

[0010] Figure 1 An exemplary perspective view of a carriage according to one or more embodiments of the present disclosure is shown;

[0011] Figure 2 A block diagram of an exemplary in-vehicle computing system of a vehicle having an object detection system according to one or more embodiments of the present disclosure is shown;

[0012] Figure 3 A data flow of an object detection system according to one or more embodiments of the present disclosure is shown;

[0013] Figure 4 An exemplary scene for object detection of an object detection system according to one or more embodiments of the present disclosure is shown;

[0014] Figure 5 An example of an existing neural network for object detection in an object detection system according to one or more embodiments of the present disclosure is shown;

[0015] Figure 6 An exemplary unauthorized modification of the results of an object detection system according to one or more embodiments of the present disclosure is shown;

[0016] Figure 7 An exemplary neural network including the YOLO algorithm for object detection in an object detection system according to one or more embodiments of the present disclosure is shown;

[0017] Figure 8 A flowchart of a method for object detection according to one or more embodiments of the present disclosure is shown;

[0018] Figure 9 A flowchart of a method for verifying the output of a neural network for object detection according to one or more embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0019] As described above, a vehicle may include object detection and perception for monitoring the vehicle's surrounding environment. For example, a vehicle, such as Figure 1 the vehicle shown, may be at least partially autonomous and may adjust one or more vehicle systems based on object detection. In other examples, a vehicle may include one or more driver assistance systems for intermittently adjusting vehicle operation. Figure 1 the vehicle shown may include an on-vehicle computing system and a vehicle control system, as Figure 2 shown. The vehicle may include sensors that may provide data to an object detection system, as shown by the sensor data stream of Figure 3 . The object detection system may localize and classify objects in a scene, such as Figure 4 the scene shown. Specifically, the object detection system may include a neural network for localizing and classifying objects, as Figure 5 shown. However, in some examples, the location and classification of detected objects may be modified by an unauthorized entity, as Figure 6 shown. Figure 7 A neural network with encryption to reduce the incidence of unauthorized modification is shown in Figure 8 . For example, the localization and classification for object detection may be encrypted according to the method of Figure 9 . A method for verifying the encryption is shown in

[0020] Figure 1 shows the interior of the passenger compartment 100 of vehicle 102 (where a driver and / or one or more passengers may sit). Vehicle 102 may be a road vehicle, as well as other types of vehicles. Specifically, Figure 1 vehicle 102 may be a motor vehicle that includes drive wheels (not shown) and an engine 104. In some examples, engine 104 may be an internal combustion engine. In other examples, engine 104 may be an electric engine, or may include hybrid components. For example, vehicle 102 may include a hybrid propulsion system that includes an energy conversion device that may be operable to absorb energy from vehicle movement and / or the engine and convert the absorbed energy into an energy form suitable for storage by an energy storage device. Vehicle 102 may include a fully electric vehicle that incorporates a fuel cell, a solar capture element, and / or other energy storage systems for powering the vehicle.

[0021] Additionally, in some examples, vehicle 102 can be an autonomous vehicle. In some examples, vehicle 102 is a fully autonomous vehicle (e.g., a fully self-driving vehicle) that is configured to travel without user input. For example, vehicle 102 can independently control vehicle systems to guide the vehicle to a desired location and can sense environmental features to guide the vehicle (e.g., such as via object detection). In some examples, vehicle 102 is a partially autonomous vehicle. In some examples, vehicle 102 can have an autonomous mode and a non-autonomous mode, in which the vehicle operates without user input in the autonomous mode and the user commands the vehicle in the non-autonomous mode. Additionally, in some examples, although the autonomous vehicle control system can primarily control the vehicle in the autonomous mode, the user can input commands to adjust vehicle operation, such as a command to change the vehicle speed, a braking command, a turning command, etc. In still some other examples, the vehicle can include at least one driver assistance system for partially controlling the vehicle, such as a cruise control system, a collision avoidance system, a lane change system, etc.

[0022] Vehicle 102 can include multiple vehicle systems, including a braking system for providing braking, an engine system for providing power to the vehicle's wheels, a steering system for adjusting the vehicle's direction, a transmission system for controlling the gear selection of the engine, an exhaust system for processing exhaust gas, etc. Additionally, vehicle 102 includes an in-vehicle computing system 109. The in-vehicle computing system 109 can include an autonomous vehicle control system for at least partially controlling the vehicle systems during autonomous driving. As an example, when operating in the autonomous mode, the autonomous vehicle control system can monitor the vehicle's surrounding environment via multiple sensors (e.g., such as cameras, radar, ultrasonic sensors, GPS signals, etc.).

[0023] As shown, the dashboard 106 may include various displays and controls accessible to a human user (also referred to as a passenger) of the vehicle 102. For example, the dashboard 106 may include user input devices such as the touchscreen 108 of the in-vehicle computing system 109, an audio system control panel, and the instrument cluster 110. The touchscreen 108 may receive user input for the in-vehicle computing system 109 to control audio output, visual display output, user preferences, control parameter selection, and the like. In some examples, the dashboard 106 may include an input device for a user to switch the vehicle between an autonomous mode and a non-autonomous mode. For example, the vehicle includes an autonomous mode in which the autonomous vehicle control system operates the vehicle at least partially independently and a non-autonomous mode in which a vehicle user operates the vehicle. The vehicle user may switch between the two modes via user input on the dashboard 106. Additionally, in some examples, the dashboard 106 may include one or more controls for the autonomous vehicle control system, such as for selecting a destination, setting a desired vehicle speed, setting navigation preferences (e.g., preference for highways over city streets), and the like. Still further, in some examples, the dashboard 106 may include one or more controls for driver assistance programs, such as a cruise control system, a collision avoidance system, and the like. Additionally, additional user interfaces not shown may be present in other parts of the vehicle, such as near at least one passenger seat. For example, the vehicle may include a row of rear seats having at least one touchscreen for controlling the in-vehicle computing system 109.

[0024] The vehicle compartment 100 may further include one or more user objects stored in the vehicle before, during, and / or after travel, such as the mobile device 128. The mobile device 128 may include a smart phone, a tablet computer, a laptop computer, a portable media player, and / or any suitable mobile computing device. The mobile device 128 may be connected to the in-vehicle computing system via a communication link 130. The communication link 130 may be wired (e.g., via Universal Serial Bus [USB], Mobile High-Definition Link [MHL], High-Definition Multimedia Interface [HDMI], Ethernet, etc.) or wireless (e.g., via Bluetooth, WIFI, WIFI Direct, Near Field Communication [NFC], cellular connection, etc.), and is configured to provide two-way communication between the mobile device and the in-vehicle computing system. The mobile device 128 may include one or more wireless communication interfaces for connecting to one or more communication links (e.g., one or more of the exemplary communication links described above). The wireless communication interface may include: one or more physical devices, such as antennas or ports, which are coupled to data lines for carrying data transmitted or received; and one or more modules / drivers for operating the physical devices according to other devices in the mobile device. For example, the communication link 130 may provide sensor and / or control signals from various vehicle systems (such as the vehicle audio system, the sensor subsystem, etc.) and the touch screen 108 to the mobile device 128, and may provide control and / or display signals from the mobile device 128 to the in-vehicle system and the touch screen 108. The communication link 130 may also provide power from the in-vehicle power supply to the mobile device 128 to charge the internal battery of the mobile device.

[0025] The vehicle computing system 109 may also be communicatively coupled to additional devices that are operated and / or accessed by a user but are located external to the vehicle 102, such as one or more external devices 150. In the depicted embodiment, the external devices are located external to the vehicle 102, but it will be appreciated that in alternative embodiments, the external devices may be located within the passenger compartment 100. The external devices may include server computing systems, personal computing systems, portable electronic devices, electronic bracelets, electronic headbands, portable music players, electronic activity tracking devices, pedometers, smart watches, GPS systems, and the like. The external device 150 may be connected to the vehicle computing system via a communication link 136, which may be wired or wireless, as discussed with reference to the communication link 130, and the external device is configured to provide two-way communication between the external device and the vehicle computing system. For example, the external device 150 may include one or more sensors, and the communication link 136 may transmit sensor outputs from the external device 150 to the vehicle computing system 109 and the touch screen 108. The external device 150 may also store and / or receive information regarding navigation map data, image feature map data, etc., and may transmit such information from the external device 150 to the vehicle computing system 109 and the touch screen 108.

[0026] The vehicle computing system 109 may analyze inputs received from the external device 150, the mobile device 128, and / or other input sources, and provide outputs via the touch screen 108 and / or the speaker 112, communicate with the mobile device 128 and / or the external device 150, and / or perform other actions based on the evaluation. In some embodiments, all or a portion of the evaluation may be performed by the mobile device 128 and / or the external device 150. In some embodiments, the external device 150 may include an in-vehicle computing device of another vehicle.

[0027] In some embodiments, one or more of the external devices 150 may be indirectly communicatively coupled to the vehicle computing system 109 via the mobile device 128 and / or another of the external devices 150. For example, the communication link 136 may communicatively couple the external device 150 to the mobile device 128 such that outputs from the external device 150 are relayed to the mobile device 128. Data received from the external device 150 may then be aggregated at the mobile device 128 with data collected by the mobile device 128, and the aggregated data may then be transmitted via the communication link 130 to the vehicle computing system 109 and the touch screen 108. Similar data aggregation may occur at the server system and may then be transmitted via the communication link 136 / 130 to the vehicle computing system 109 and the touch screen 108.

[0028] Figure 2A block diagram of an in-vehicle computing system 109 configured and / or integrated within a vehicle 102 is shown. The in-vehicle computing system 109 may execute one or more of the methods described herein in some embodiments. The in-vehicle computing system may include or be coupled to various vehicle systems, subsystems, hardware components, and software applications and systems integrated in or capable of being integrated in the vehicle 102 to enhance the in-vehicle experience of the driver and / or passengers. Additionally, the in-vehicle computing system may be coupled to a system for providing autonomous vehicle control.

[0029] The in-vehicle computing system 109 may include one or more processors, the one or more processors including an operating system processor 214 and an interface processor 220. The operating system processor 214 may execute an operating system on the in-vehicle computing system and control the input / output, display, playback, and other operations of the in-vehicle computing system. The interface processor 220 may interface with a vehicle control system 230 via an in-vehicle system communication module 224.

[0030] The in-vehicle system communication module 224 may output data to the vehicle control system 230 while also receiving data inputs from other vehicle components and systems, for example, via the vehicle control system 230. When outputting data, the in-vehicle system communication module 224 may provide a signal corresponding to an output of any state of the vehicle, the vehicle's surrounding environment, or any other information source connected to the vehicle via a bus. Vehicle data outputs may include, for example, analog signals (such as current speed), digital signals provided by separate information sources (such as clocks, thermometers, position sensors such as global positioning system [GPS] sensors, inertial measurement systems [IMS]), and digital signals propagated through vehicle data networks (such as an engine controller area network [CAN] bus by which engine-related information may be transmitted, a climate control CAN bus by which climate control-related information may be transmitted, and a multimedia data network by which multimedia data may be transmitted between multimedia components in the vehicle). For example, vehicle data outputs may be output to the vehicle control system 230, and the vehicle control system 230 may adjust vehicle controls 236 based on the vehicle data outputs. For example, the in-vehicle computing system 109 may retrieve the current speed of the vehicle estimated by a wheel sensor from the engine CAN bus, the power state of the vehicle via the vehicle's battery and / or power distribution system, the ignition state of the vehicle, etc. Additionally, other interfacing methods such as Ethernet may also be used without departing from the scope of the present disclosure.

[0031] A storage device 208 may be included in the vehicle computing system 109 to store data in a non-volatile form, such as instructions executable by processors 214 and 220. The storage device 208 may store application data, including pre-recorded sounds, to enable the vehicle computing system 109 to run applications to connect to a cloud-based server and / or collect information for transmission to a cloud-based server. The applications may retrieve information collected by vehicle systems / sensors, input devices (e.g., user interface 218), data stored in volatile memory 219A or non-volatile storage devices (e.g., memory) 219B, devices communicating with the vehicle computing system (e.g., mobile devices connected via a Bluetooth link), etc. The vehicle computing system 109 may also include volatile memory 219A. The volatile memory 219A may be random access memory (RAM). Non-transitory storage devices, such as non-volatile storage device 208 and / or non-volatile memory 219B, may store instructions and / or code that, when executed by a processor (e.g., operating system processor 214 and / or interface processor 220), control the vehicle computing system 109 to perform one or more of the actions described in the present disclosure.

[0032] One or more additional sensors may be included in the sensor subsystem 210 of the vehicle computing system or 109. For example, the sensor subsystem 210 may include multiple sensors for monitoring the vehicle's surrounding environment. For example, the sensor subsystem 210 may include multiple cameras 225, one or more radars 226, one or more lidars 227, and one or more ultrasonic sensors 228. For example, the sensors of the sensor subsystem 210 may be used for object detection, such as through an object detection system 232. The sensor subsystem 210 of the vehicle computing system 109 may communicate with various vehicle sensors and receive inputs from various vehicle sensors and may also receive user inputs. For example, the inputs received by the sensor subsystem 210 may include shift gear position, transmission clutch position, throttle pedal input, brake input, transmission selector position, vehicle speed, engine speed, mass air flow through the engine, ambient temperature, intake temperature, etc., and inputs from climate control system sensors (such as heat transfer fluid temperature, antifreeze temperature, fan speed, passenger compartment temperature, desired passenger compartment temperature, ambient humidity, etc.), inputs from audio sensors detecting voice commands issued by a user, inputs from remote key sensors that receive commands from the vehicle's remote key and optionally track the geographical location / proximity of the remote key. While some vehicle system sensors may communicate with the sensor subsystem 210 individually, other sensors may communicate with both the sensor subsystem 210 and the vehicle control system 230, or may communicate indirectly with the sensor subsystem 210 via the vehicle control system 230.

[0033] The in-vehicle computing system 200 may include a microphone 202 to measure ambient noise in the vehicle, measure ambient noise outside the vehicle, and the like. One or more additional sensors may be included in and / or communicatively coupled to the sensor subsystem 210 of the in-vehicle computing system 200. For example, the sensor subsystem 210 may include and / or be communicatively coupled to cameras, such as a rear-view camera for assisting a user in parking the vehicle, a cabin camera for identifying the user, and / or a front-view camera for evaluating the quality of a forward road segment. The aforementioned cameras may also be used to provide images to a computer vision-based traffic sign detection module. The sensor subsystem 210 of the in-vehicle computing system 200 may communicate with various vehicle sensors and receive inputs from various vehicle sensors and may also receive user inputs. While some vehicle system sensors may communicate with the sensor subsystem 210 individually, other sensors may communicate with both the sensor subsystem 210 and the vehicle control system 230, or may communicate with the sensor subsystem 210 indirectly via the vehicle control system 230. The sensor subsystem 210 may serve as an interface (e.g., a hardware interface) and / or a processing unit for receiving and / or processing signals received from one or more of the sensors described in this disclosure.

[0034] The navigation subsystem 211 of the in-vehicle computing system 109 may generate and / or receive navigation information, such as location information (e.g., via the GPS / IMS sensor 204 and / or other sensors from the sensor subsystem 210), route guidance, traffic information, point of interest (POI) identification, and / or provide other navigation services to the user. The navigation subsystem 211 may include an input / output port 280, including an analog-to-digital converter, a digital input, a digital output, a network output, a radio frequency transmitter, and the like. In some examples, the navigation subsystem 211 may interface with the vehicle control system 230.

[0035] The external device interface 212 of the in-vehicle computing system 109 may be capable of being coupled to and / or communicating with one or more external devices 150 located outside the vehicle 102. Although the external devices are depicted as being located outside the vehicle 102, it will be understood that they may be temporarily housed within the vehicle 102, such as when a user is operating an external device while operating the vehicle 102. In other words, the external devices 150 are not integral with the vehicle 102. The external devices 150 may include a mobile device 128 (e.g., connected via Bluetooth, NFC, WIFI Direct, or other wireless connection) or alternatively a Bluetooth-enabled device 252. The mobile device 128 may be a mobile phone, smartphone, wearable device / sensor, or other portable electronic device that can communicate with the in-vehicle computing system via wired and / or wireless communication. Other external devices include an external service 246. For example, the external devices may include an out-of-vehicle device that is separate from and located outside the vehicle. Other external devices include an external storage device 254, such as a solid-state drive, pen drive, USB drive, etc. Without departing from the scope of the present disclosure, the external devices 150 may communicate with the in-vehicle computing system 109 wirelessly or via a connector. For example, the external devices 150 may communicate with the in-vehicle computing system 109 via the external device interface 212 via a network 260, a Universal Serial Bus (USB) connection, a direct wired connection, a direct wireless connection, and / or other communication links.

[0036] The external device interface 212 may provide a communication interface to enable the in-vehicle computing system to communicate with a mobile device associated with a user's contacts. For example, the external device interface 212 may enable the establishment of a voice call with and / or the sending of a text message (e.g., SMS, MMS, etc.) to a mobile device 128 associated with a user's contacts (e.g., via a cellular communication network). Additionally, in some examples, a vehicle user may adjust autonomous vehicle operation via an application on a mobile device 128 associated with the user. The external device interface 212 may additionally or alternatively provide a wireless communication interface to enable the in-vehicle computing system to synchronize data with one or more devices (e.g., the user's mobile device) in the vehicle via WIFI Direct.

[0037] One or more applications 248 may be operable on an external service 246. As an example, an external service application 248 may be operable to aggregate and / or analyze data from multiple data sources. For example, the external service application 248 may aggregate data from one or more social media accounts of a user, data from an in-vehicle computing system (e.g., sensor data, log files, user input, etc.), data from Internet queries (e.g., weather data, POI data), etc. The collected data may be transmitted to another device and / or analyzed by an application to determine the context of the user, vehicle, and environment and perform actions based on the context (e.g., request / send data to other devices).

[0038] The in-vehicle computing system 109 may also include an antenna 206. The antenna 206 is shown as a single antenna, but in some embodiments may include one or more antennas. The in-vehicle computing system may obtain broadband wireless Internet access via the antenna 206 and may also receive broadcast signals such as radio, television, weather, traffic, etc. The in-vehicle computing system may receive location signals such as GPS signals via one or more antennas 206. The in-vehicle computing system may also receive wireless commands via FR (such as via the antenna 206) or via infrared or other means through a suitable receiving device. For example, the antenna 206 may receive a voice call (e.g., a phone call). Additionally, the antenna 206 may provide an AM / FM radio signal to an external device 150 (such as to a mobile device 128) via the external device interface 212.

[0039] The vehicle control system 230 may include vehicle controls 236 for controlling various aspects of the vehicle systems. For example, the vehicle controls 236 include a steering control system 238, a braking control system 240, and an acceleration control system 242. The vehicle controls 236 may include additional control systems. In some examples, the vehicle controls 236 may operate autonomously, such as during autonomous vehicle operation. In other examples, the vehicle controls 236 may be controlled by a user. Additionally, in some examples, the user may primarily control the vehicle controls 236 while various driver assistance programs may intermittently adjust the vehicle controls 236 to improve vehicle performance. For example, various driver assistance systems may include a cruise control system, a lane departure warning system, a collision avoidance system, an adaptive braking system, etc.

[0040] The braking control system 240 can be configured to control the amount of braking force applied to the vehicle. For example, during non-autonomous operation mode, the braking system 240 can be controlled by a brake pedal. For example, a user can depress the brake pedal to increase the amount of braking applied to the vehicle. During autonomous operation mode, the braking system 240 can be autonomously controlled. For example, the vehicle control system 230 can determine that additional braking is requested and can apply additional braking. In some examples, the autonomous vehicle control system can depress the brake pedal to apply braking (e.g., to reduce the vehicle speed and / or stop the vehicle). In some examples, the driver assistance system can adjust the braking control system 240.

[0041] The acceleration control system 242 can be configured to control the amount of acceleration applied to the vehicle. For example, during non-autonomous operation mode, the acceleration control system 242 can be controlled by an acceleration pedal. For example, a user can depress the acceleration pedal to increase the amount of torque applied to the vehicle wheels, thereby accelerating the vehicle. During autonomous operation mode, the acceleration control system 242 can be controlled by the vehicle control system 230. In some examples, the driver assistance system can adjust the acceleration control system 242. For example, the vehicle control system 230 can determine that additional vehicle speed is requested and can increase the vehicle speed via acceleration. In some examples, the vehicle control system 230 can depress the acceleration pedal to accelerate the vehicle. As an example of the driver assistance system adjusting the acceleration control system 242, the driver assistance system can be a cruise control system and can include adjusting the vehicle acceleration to maintain a desired speed during vehicle operation.

[0042] The steering control system 238 can be configured to control the direction of the vehicle. For example, during non-autonomous operation mode, the steering control system 238 can be controlled by a steering wheel. For example, a user can turn the steering wheel to adjust the vehicle direction. During autonomous operation mode, the steering control system 238 can be controlled by the vehicle control system 230. In some examples, the driver assistance system can adjust the steering control system 238. For example, the vehicle control system 230 can determine that a change in vehicle direction is requested and can change the vehicle direction by controlling the steering control system 238. For example, the vehicle control system 230 can adjust the axles of the vehicle to change the vehicle direction.

[0043] The vehicle control system 230 may also include: controls for adjusting the settings of various vehicle controls (or vehicle system control elements) related to the engine and / or auxiliary elements within the vehicle cabin, such as steering wheel controls (e.g., steering wheel-mounted audio system controls, climate control, cruise control, windshield wiper controls, headlight controls, turn signal controls, etc.); dashboard controls; microphones; gear shifters; door / window controls located in the driver's door or passenger door; seat controls; cabin light controls, etc. The vehicle controls may also include internal engine and vehicle operation controls (e.g., engine controller modules, actuators, valves, etc.), which are configured to receive instructions via the vehicle's CAN bus to change the operation of one or more of the engine, exhaust system, transmission, and / or other vehicle systems. The control signal may also control a vehicle audio system (not shown). For example, the control signal may adjust audio output characteristics such as volume, equalization, audio imaging (e.g., an audio signal configuration that produces an audio output that appears to the user to originate from one or more defined locations), audio distribution among multiple speakers, etc. Similarly, the control signal may control the vents, air conditioning, and / or heater of the climate control system. For example, the control signal may increase the delivery of cool air to a specific section of the cabin. For example, when operating in autonomous mode, the autonomous vehicle control system may control some or all of the above vehicle controls. In addition, the vehicle control system 230 may include multiple driver assistance systems, such as a cruise control system and a collision avoidance system.

[0044] Control elements located outside the vehicle (e.g., controls for a security system) may also be connected to the computing system 109, such as via the communication module 224. The control elements of the vehicle control system may be physically and permanently located on and / or within the vehicle for receiving user input. In addition to receiving control instructions from the on-vehicle computing system 109, the vehicle control system 230 may also receive input from one or more external devices 150 operated by the user (such as from the mobile device 128). This allows aspects of the vehicle controls 236 to be controlled based on user input received from the external device 150.

[0045] The vehicle control system 230 includes an object detection system 232 for detecting objects. For example, the object detection system 232 can receive sensor data from the sensor subsystem 210 via the in-vehicle system communication module 224, and can identify objects in the vehicle's surrounding environment, such as traffic lights, other vehicles, pedestrians, etc. The output of the object detection system 232 can be used in a variety of systems, such as for adjusting vehicle controls 236, for notifying the user of an object, for autonomous vehicle control, for driver assistance systems, etc. Specifically, the object detection system 232 includes a neural network 234. The neural network 234 can be a convolutional neural network (CNN) trained on sensor data to detect and identify objects. As an example, the object detection system 232 can employ YouOnly Look Once (YOLO) as a type of neural network object detector for detecting and identifying objects via the neural network 234. In other examples, the object detection system 232 can use other object detectors, such as Spatial Pyramid Pooling (SPP), Fast R-CNN (FRCN), Region Proposal Network (RPN), Single Shot Detector (SSD), Deconvolutional Single Shot Detector (DSSD), RetinaNet, Deformable Convolutional Network, etc. Fully Convolutional Network (FCN), Batch Normalization (BN), deconvolutional layers, etc. are exemplary internal sub-components within a neural network, such as for example the neural network 234. The object detection system 232 will be described herein with respect to the YOLO detector. However, without departing from the scope of the current disclosure, other object detection frameworks (more appropriately, object detectors), such as the above object detectors, can be used. Object detection systems, such as the object detection system 232, are elaborated in more detail below with respect to Figures 3 to 9 be elaborated in more detail.

[0046] Next, Figure 3 is shown Figure 1 and Figure 2 a block diagram of an exemplary data flow 300 of the object detection system 232 of the in-vehicle computing system 109. For example, Figure 3 the exemplary data flow 300 can be used for Figure 2 the object detection system 232 in the in-vehicle computing system 109. Similar components may have the same numbering and will not be introduced again. As Figure 3As shown, data from multiple sensors can be transmitted to the object detection system 232. For example, data from the GPS / IMS 204 and the sensor 210 can be transmitted to the object detection system 232. The object detection system 232 includes multiple sensor processing blocks for processing sensor data. As shown, data from the GPS / IMS 204 and the camera 225 are transmitted to the sensor processing block 312, data from the radar 226 are transmitted to the sensor processing block 314, data from the lidar 227 are transmitted to the sensor processing block 316, and data from the ultrasonic sensor 228 are transmitted to the sensor processing block 318. Data from each of the sensor processing block 312, the sensor processing block 314, the sensor processing block 316, and the sensor processing block 318 are transmitted to the sensor fusion block 322. For example, the sensor fusion block 332 can combine data from each sensor processing block. The sensor fusion 322 can also incorporate vehicle-to-vehicle (V2V) and / or vehicle-to-infrastructure (V2I) 320 and map data 324. Data from the sensor fusion 322 can be used for object detection, or object detection can be used for the data that includes the data fusion 322, depending on the type of sensor fusion technology selected. Data from the sensor fusion 322 can be shared with the neural network 234 for object detection. For example, images from the camera 225 and background data from the GPS / IMS 204 and other sensors can be provided as inputs to the neural network 234. For example, camera images can be combined with background data from GPS, radar, lidar, ultrasonic, and other sensors, which can improve the accuracy of the neural network during object detection. The neural network 234 is described in more detail with respect to Figure 7 The output of the neural network 234 can be the position and classification of an object in the camera image from the camera 225. For example, the neural network 234 can output a bounding box of an object in the camera image from the camera 225, which describes the position and size of the detected object. The output of the sensor fusion 322, the output of the neural network 234, and the driver state 326 can be transmitted to the action engine 328. For example, the action engine 328 can determine how to adjust the vehicle controls based on the results of the sensor fusion block 322 and the driver state 326 (e.g., steering wheel position, accelerator pedal position, etc.). Finally, the results of the action engine 328 are used to adjust the vehicle controls 236. For example, the results of the object detection system 232 can be used by one or both of the autonomous vehicle control system and the driver assistance system.

[0047] Next, Figure 4Shows an exemplary image 400 with object detection. Specifically, the example image 400 can be an image from a vehicle's camera, and this image can be used for object detection. In some examples, additional sensor data such as radar, lidar, ultrasonic sensors, maps, etc. can be used to enhance object detection. As shown, the image shows multiple objects, and each detected object is identified by a bounding box. For example, Figure 4 The multiple shown bounding boxes can be the output of the neural network of the object detection system, such as Figure 2 The object detection system 232 shown. For example, image 400 includes a first vehicle 418, a second vehicle 420, a traffic signal 422, a pedestrian 424, a truck 426, a first road 414, and a second road 416. Additionally, the detected objects can be identified by the bounding boxes. Each bounding box can locate the object and can also include classification information of the detected object. The classification information can include the type of the identified object. As shown, the first vehicle 418 is identified via the bounding box 404, and the bounding box can include the position of the first vehicle 418 and the classification of the first vehicle 418. Additionally, the second vehicle 420 can be identified via the bounding box 406. The traffic signal 422 can be identified via the bounding box 408, the pedestrian 424 can be identified via the bounding box 410, and the truck 426 can be identified via the bounding box 412. In some examples, the bounding boxes can be used by the vehicle system to adjust vehicle operations, such as adjusting the vehicle direction and speed based on the detected objects (e.g., the objects identified by the bounding boxes).

[0048] Specifically, each bounding box can be described by a set of coordinates [X,Y,W,H,P,C], which can describe the position of the bounding box, the size of the bounding box, the probability of the detected object, and the category of the detected object. For example, the coordinates [X,Y] can correspond to the position of the bounding box in the image relative to the Cartesian axis 499. For example, both the bounding box 406 and the bounding box 404 can have the same Y coordinate and different X coordinates. Additionally, the coordinates [W,H] can correspond to the width and height of the bounding box. For example, the size of the bounding box is based on the size of the detected object, so the width and height of the bounding box can correspond to the size of the detected object. Additionally, the coordinate P refers to the probability that the detected object exists. Furthermore, the coordinate C refers to the category assigned to the detected object. As an example, the category C associated with the bounding box 410 can be the pedestrian category, while the category C associated with the bounding box 404 can be the vehicle category.

[0049] Next, Figure 5 Shows an exemplary architecture of a neural network 500, such as can be used for object detection. For example, the neural network 500 can be Figure 2 and Figure 3 The neural network 234 of. Specifically, Figure 5Shows a neural network (e.g., YOLO neural network) with the YOLO algorithm for object detection. For example, the YOLO neural network includes multiple convolutional layers, followed by two fully connected layers. For example, the image 502 can be input into the feature extraction block 504. For example, the feature extraction block 504 can use deep learning to extract features (e.g., such as objects) from the image 502. Specifically, the feature extraction block 504 includes multiple convolutional layers of the neural network. The output of the feature extraction block 504 can be the localization head 506 and the classification head 508. The localization head 506 can describe the location of the extracted features (e.g., detected objects), while the classification head 508 can describe the object class. Specifically, the localization head 506 and the classification head 508 can be part of the fully connected layers of the neural network. For example, the localization head 506 includes a first plurality of nodes of the fully connected layer of the neural network, and the classification head 508 includes a second plurality of nodes of the fully connected layer of the neural network. The output of the localization head 506 can be the localization output 510, which can include the coordinates [X, Y, W, H, P] of the detected object. For example, the coordinates [X, Y] can describe the location of the bounding box identifying the object, and the coordinates [W, H] can describe the size of the bounding box. In addition, the coordinate [P] can describe the probability of the detected object. The output of the classification head 508 is the classification output 512, including the class C of the detected object. For example, the class C can describe the type of the detected object, such that each possible value of the class C corresponds to an object type, such as a car, a pedestrian, a truck, a traffic light, a traffic sign, etc. In addition, the classification output 512 and the localization output 510 are the outputs of the fully connected layers of the neural network. In summary, the localization output 510 and the classification output 512 can describe the location and size of the bounding box of the object, the class of the detected object, and the probability of the object's existence. In this way, the output of the YOLO neural network can detect objects and output the bounding box coordinates of the objects and the classification of the objects.

[0050] However, in some examples, the results of a neural network architecture such as neural network 500 may be modified by an unauthorized entity. Specifically, unauthorized modifications may occur at stage 514 such that the output of one or both of localization head 506 and classification head 508 is modified. Stage 514 occurs in the fully connected layers of the neural network where classification and localization occur. As an example, an unauthorized modification of stage 514 may occur in the fully connected layers of the neural network, i.e., the layers where the neural network classifies and localizes objects in an image. For example, one or both of the first plurality of nodes (e.g., corresponding to localization head 506) and the second plurality of nodes (e.g., corresponding to classification head 508) may be targeted for unauthorized modification. The unauthorized modification can be an intentional modification by an external entity and can adjust at least one output of the neural network, such as bounding box size, location, probability, or object class. Additionally, by modifying the output of YOLO neural network 500, vehicle operation can be adjusted. The unauthorized modification may reduce the accuracy of the object detection system, may reduce the functional safety of the vehicle, and may reduce customer satisfaction with vehicle systems such as autonomous vehicle control systems and driver assistance systems.

[0051] Next, Figure 6 A view 600 of possible unauthorized modifications that may occur in the neural network of an object detection system (e.g., Figure 2 the object detection system 232) is shown. For example, the neural network can be a YOLO neural network, the architecture of which is shown in Figure 5Shown in. As a first example, a located object bounding box 602 can be described by a set of coordinates [X1, Y1, W1, H1] that describe the size and position of the bounding box 602. The located object bounding box 602 can be the output of a localization node of a YOLO neural network. An unauthorized modification 604 can modify the located object bounding box 602 to a modified object bounding box 606 described by a set of adjusted coordinates [X2, Y2, W2, H2]. For example, the set of adjusted coordinates [X2, Y2, W2, H2] may be different from the coordinates [X1, Y1, W1, H1]. As a second example, a located object presence probability 608 can be described by a set of coordinates [X1, Y1, W1, H1, P = 1]. The located object presence probability 608 can describe the probability [P = 1] that a detected object exists and can be the output of a localization node of a YOLO neural network. An unauthorized modification 610 can adjust the located object presence probability 608 to a modified object presence probability 612 with an adjusted set of coordinates [X1, Y1, W1, H1, P = 0]. Specifically, the adjusted set of coordinates includes the adjusted probability of the object (e.g., a probability of zero instead of one). As a third example, a classified object class 614 includes the coordinates [X1, Y1, W1, H1, P = 1, class = pedestrian]. For example, the object class [class = pedestrian] classifies the type of the detected object, which can be used to adjust vehicle operation. An unauthorized modification 616 can adjust the classified object class 614 to a modified object class 618 such that the coordinates [X1, Y1, W1, H1, P = 1, class = pedestrian] are replaced by [X1, Y1, W1, H1, P = 1, class = car]. Specifically, the class of the object is modified from "pedestrian" to "car".

[0052] Such unauthorized modifications may trigger unrequested behaviors of the vehicle, such as turning to avoid non-existent objects, not avoiding upcoming objects, incorrectly responding to traffic signals, etc., thereby reducing user comfort during vehicle operation. For example, the vehicle may be operated differently when a pedestrian is detected in the road compared to when a vehicle is detected in the road. For example, unauthorized modifications may cause an increase in the incidence of noise, vibration, and harshness (NVH) problems. As another example, unauthorized modifications may change the vehicle route, thereby increasing the amount of driving time. Overall, unauthorized modifications to the output of the YOLO algorithm may reduce the safety of the vehicle and / or reduce customer satisfaction.

[0053] The inventors have recognized herein that including a cryptographic mechanism in the fully connected layer of a neural network can reduce the incidence of unauthorized modifications and can allow a controller to detect unauthorized modifications. For example, including a cryptographic mechanism can improve the accuracy of vehicle control and enhance customer satisfaction and vehicle safety. For example, a cryptographic mechanism can be added inside the nodes (neurons) of the fully connected layer to verify the encrypted output. Such a cryptographic mechanism may involve using multiple private and public keys and verifying signatures before accepting the results of an object detection system.

[0054] Next, Figure 7 FIG. shows a schematic diagram of a YOLO neural network 700 for object detection. For example, the YOLO neural network 700 may be similar to Figure 5 the YOLO neural network shown. However, the YOLO neural network 700 includes a cryptographic mechanism to prevent unauthorized modification of the output of the YOLO neural network 700. In other words, the YOLO neural network 700 is a YOLO neural network with cryptographic protection. As Figure 7 shown, the YOLO neural network 700 includes a first convolutional layer 702, a first max pool 704, additional convolutional and pooling layers 706 (e.g., additional convolutional layers 2-23 according to the architecture of the YOLO neural network). The YOLO neural network 700 includes multiple layers into which a camera image can be input. The convolutional layers of the neural network can process the input data and can output the location of the bounding box and the classification of the object. For example, the input is provided to the first convolutional layer 702, and the output of the first convolutional layer 702 is provided to the first max pooling layer 704. The output of the first max pooling layer 704 is passed through the additional convolutional and pooling layers 706. Next, the output of the additional convolutional layer 706 is passed to the convolutional layer 24 708, which outputs to the first fully connected layer 710.

[0055] Regarding the second fully connected layer 712 of the YOLO neural network, according to some embodiments, the following three mentioned concepts apply. As the first mentioned concept, in some embodiments, the system makes modifications at the second fully connected layer 712 of YOLO. The second fully connected layer may include only the output layer of the YOLO neural network or the last layer of the YOLO neural network or the last fully connected layer of the YOLO neural network. Neural network nodes may perform the following operations: a) First calculate ΣWX + B (for the weight matrix 718), where X is the output of the nodes in the previous layer, W is the weight connecting the nodes in the previous layer to the nodes in the current layer, and B is the bias of the current node; and b) Then apply the activation function "f" to ΣWX + B to insert non-linearity, or the output of the node = f(ΣWX + B) (such as, for example, the activation function 736). Any layer of the neural network may include an array of neurons of the above kind. Similarly, the fully connected layer 2 of the YOLO neural network may be an array of neurons of the above kind, which performs f(ΣWX + B), where X = the output of the fully connected layer 1, W = the weight connecting the neurons of the fully connected layer 1 to the fully connected layer 2, f = the activation function, and B = the bias of the fully connected layer 2.

[0056] As the second mentioned concept, in some embodiments, the system applies cryptographic components in f(ΣWX + B) to all nodes of the fully connected layer 2 of the YOLO neural network. In the array of all nodes of the fully connected layer 2, some nodes contain localization nodes and the remaining nodes contain classification nodes. The modification inside each neuron of the fully connected layer 2 preferably includes: a) finding ΣWX + B, where X = the output of the fully connected layer 1, W = the weight connecting the neurons of the fully connected layer 1 to the fully connected layer 2, B = the bias of the fully connected layer 2, where Y1 = (ΣWX + B) (localization node) and Y2 = (ΣWX + B) (classification node); b) adding cryptographic components to the nodes of the fully connected layer 2, the cryptographic components including a signature for each localization node and a signed encryption 732 for each classification node 716 of the fully connected layer 2, thus providing Sign(Y1) 720 for the localization node 714 and Sign(Y2') 734 for the classification node 716, where Y2' = encryption(Y2) 732; and c) applying the activation function to all generated Sign(Y1) 720 and Sign(Y2') 734.

[0057] As a third mentioned concept, in some embodiments, the combination of steps a), b) and c) from the second mentioned concept includes a fully connected layer 2, which may only include an array of nodes, some of which are localization nodes and the remaining nodes are classification nodes. Thus, at each node, the fully connected layer 2 may be equal to ((ΣWX + B) + cryptographic component + activation function). Alternatively, at each node, the fully connected layer 2 may be equal to activation_function(cryptographic_component(ΣWX + B)). Alternatively, at each node, the fully connected layer 2 may be equal to f(Crypto(ΣWX + B)), where f is the activation function and Crypto is the added cryptographic component. Operations a), b) and c) from the second mentioned concept above preferably occur within each node. In some embodiments, the system includes a modification to the node functionality of the fully connected layer 2 to insert a cryptographic component, where the cryptographic component is the described signature (on the localization nodes) and signed encryption (on the classification nodes).

[0058] Still referring to Figure 7 , in some embodiments, the output of the first fully connected layer 710 is passed to the second fully connected layer 712. For example, the second fully connected layer 712 includes localization nodes 714 and classification nodes 716. For example, the localization nodes 714 and classification nodes 716 output a weight matrix 718, which is defined by the formula ΣWX + B described and defined above. For example, the output of the second fully connected layer 712 is the bounding box and classification category of the detected object. Additionally, to increase system security and reduce the incidence of unwanted modifications, it is preferable to apply the first, second and third mentioned concepts above. In some embodiments, the output of the node includes the final encrypted localization and classification outputs, including respectively Figure 7 738 and 740 in. As shown, SIGN(Y1) 720 includes a cryptographic signature generated based on the first private key 722. For example, the first private key 722 is stored in a replay protected memory block (RPMB) 724 and is used to generate the cryptographic signature. Additionally, SIGN(Y2') 734 includes an encryption at 732 such that Y2 is transformed into Y2'. Y2' is an encrypted version of Y2. For example, Y2 is encrypted based on the public key 728, which is stored in the write protected memory (WPM) 726 of the controller, resulting in Y2'. Additionally, Y2' includes a cryptographic signature, which is generated based on the second private key 730. The second private key 730 is stored in the RPMB 724. Thus, the result at block 734 is SIGN(Y2'), i.e., the encrypted Y2 (e.g., Y2') signed with a cryptographic signature.

[0059] In some embodiments, the keys used include symmetric key pairs. For example, the key pair (private key 1, public key 1) may be associated with the localization header signature; the key pair (private key 2, public key 2) may be associated with the classification header signature; and the key pair (private key 3, public key 3) may be associated with the encryption and decryption of the classification header. Preferably, all private and public key pairs are managed by the vehicle manufacturer.

[0060] Next, as Figure 7 shown, SIGN(Y1) from block 720 and SIGN(Y2') from block 734 are input into the leaky rectified linear unit (ReLU) activation function 736. For example, the output of the leaky ReLU activation function 736 is the cryptographically signed localization output 738 and the cryptographically signed classification output 740. For example, the cryptographically signed localization output 738 includes the encrypted coordinates [X, Y, W, H, P], and the cryptographically signed classification output 740 includes the encrypted class [C]. In other words, the classification output is both encrypted and cryptographically signed, which can reduce the incidence of unauthorized modification. Therefore, Figure 7 the YOLO architecture shown includes encryption of the classification header such that the output of the YOLO neural network includes the signed localization output 738 and the signed and encrypted classification output 740, which can reduce the incidence of unauthorized modification. In addition, since the first private key 722 and the second private key 730 are stored in the RPMB 724, neither the first private key 722 nor the second private key 730 can be modified by an external agent. For example, the cryptographically signed localization output 738 is the cryptographically signed coordinates [X, Y, W, H, P], and the cryptographically signed classification output 740 is the cryptographically signed and encrypted class C. The output of the YOLO neural network (the cryptographically signed localization output 738 and the cryptographically signed classification output 740) can be verified and decrypted via the verification method described in detail below regarding Figure 9

[0061] Next, Figure 8 illustrates a method for object detection using a cryptographically protected YOLO neural network (e.g., the YOLO algorithm) in a vehicle. Method 800 will be described with respect to the Figure 7 YOLO neural network described. In addition, the vehicle can be the vehicle 102 described with respect to Figure 1 and Figure 2 described. For example, the vehicle includes a controller and an object detection system, and the object detection system includes the Figure 7 YOLO neural network 700. The instructions for performing method 800 can be stored in the non-transitory memory of the in-vehicle computing system (e.g., the storage device 208 shown in Figure 2 ). Thus, it can be processed by a processor (e.g., Figure 2The operating system processor 214) executes method 800 based on the stored instructions and in combination with signals received from sensors of the vehicle system (such as the sensors described above with reference to Figure 2 the sensors described herein).

[0062] In step 802, method 800 includes inputting a video frame or image into the YOLO algorithm. For example, the input video frame or image can be the output of a vehicle camera, such as Figure 2 the camera 225. In some examples, a single video frame is input into the YOLO algorithm. In other examples, background information can also be input into the YOLO algorithm, such as map data, lidar data, radar data, ultrasonic sensor data, etc. For example, as Figure 3 shown, additional sensor data can be combined via sensor fusion and provided as input to the neural network. For example, by including background data, the YOLO algorithm can more accurately detect objects near the vehicle, which can improve customer satisfaction.

[0063] In step 804, method 800 includes processing the video frame or image via convolutional layers and pooling layers, as shown above in Figure 7 . For example, the video frame / image together with the background data can be passed to the convolutional layers and pooling layers, as Figure 7 shown. For example, each layer of the neural network includes a plurality of nodes, which can be used to extract object features from the camera frame / image. In addition, the last fully connected layer can include a localization head (e.g., Figure 7 the localization node 714) and a classification head (e.g., such as Figure 7 the classification node 716) for predicting the position of the detected object and the class of the detected object.

[0064] In step 806, method 800 includes generating a cryptographically signed localization output and a cryptographically signed and encrypted classification output, as shown above in Figure 7 . For example, SIGN(Y1) 720 can include a cryptographic signature based on a private key (e.g., the first private key 722 stored in Figure 7 the RPMB 724). In addition, Y2' 732 can be encrypted via a public key stored in the WPM (e.g., the public key 728 stored in Figure 7 the WPM 726). In addition, SIGN(Y2') 734 can be signed via a cryptographic signature based on a second private key stored in the RPMB (e.g., Figure 7the second private key 730 and the RPMB 724). Additionally, the signed output of the localization node and the signed / encrypted output of the classification node can be processed by an activation function (e.g., the leaky ReLU activation function 736), which can produce a localization output and a classification output at the output of the fully connected layer 2. For example, the localization output is signed with a cryptographic signature and includes coordinates [X,Y,W,H,P], and the classification output is signed with a cryptographic signature and encrypted, and includes the class C. For example, without accessing the first and second private keys stored in the RPMB, the cryptographic signatures of the localization output and the classification output cannot be replicated.

[0065] At step 808, method 800 includes applying a threshold to remove unwanted bounding boxes. For example, a threshold can be applied to remove bounding boxes that are determined not to correspond to a detected object. In some embodiments, to make a final prediction, method 800 retains those boxes with a high box confidence score (greater than 0.25) as the final prediction. The confidence score reflects the likelihood (objectness) that the box contains an object and the accuracy of the bounding box.

[0066] At step 810, method 800 includes verifying the signed localization output and the signed and encrypted classification output, as shown below in Figure 9 For example, Figure 9 the method includes determining whether the output of the YOLO neural network has not been modified and, if it is determined that the output of the YOLO neural network is valid, decrypting the classified output. For example, if the output of the fully connected layer has been modified by an unauthorized entity, the cryptographic signature may not be verifiable. If the cryptographic signature is verified, the output of the neural network can be accepted and used to adjust vehicle operation. Method 800 may then end.

[0067] Next, Figure 9 illustrates a method for verifying and decrypting the output of a YOLO neural network for object detection in a vehicle. Method 900 will be described with respect to Figure 7 the YOLO neural network 700. Additionally, the vehicle can be the vehicle 102 described with respect to Figure 1 and Figure 2 For example, the vehicle includes a controller and an object detection system that includes Figure 7 the YOLO neural network. The instructions for performing method 900 can be stored in the non - transitory memory of the in - vehicle computing system (e.g., the storage device 208 shown in Figure 2 ). Thus, the method 900 can be executed by a processor (e.g., the operating system processor 214 of Figure 2 ) based on the stored instructions and in combination with signals received from sensors of the vehicle system (such as the sensors described above with respect to Figure 2 ).

[0068] In step 902, method 900 includes using the cryptographically signed localization output as Figure 7 the output of the object detection algorithm shown. For example, the cryptographically signed localization output at 902 can be Figure 7 the cryptographically signed localization output 738, and can be a cryptographically signed output [X, Y, W, H, P] having a description of the location of the detected object and the probability of the detected object.

[0069] In step 904, method 900 includes verifying the cryptographically signed localization output of step 902 using a first public key (e.g., signature verification) (shown as "public key 1" in Figure 9 ). For example, the signature verification can be based on the public key stored in the WPM and can be used to determine whether the cryptographic signature is valid.

[0070] In step 906, method 900 includes determining whether the signature verification in step 904 was successful. For example, if the controller determines in step 904 that the cryptographic signature is valid, then the signature verification is successful and the controller determines that an unauthorized modification of the coordinates [X, Y, W, H, P] did not occur. Additionally, if the controller determines in step 904 that the cryptographic signature is invalid, then the signature verification is not successful.

[0071] If method 900 determines that the signature verification was not successful, then method 900 proceeds to step 908 and includes determining that a bounding box and object probability attack may have occurred. For example, an unauthorized modification may have been made to one or more of the bounding box coordinates, the bounding box size, and the object probability (e.g., coordinates X, Y, W, H, P). Due to the unauthorized modification, the controller can determine not to use the output of the object detection system to adjust vehicle operation. In some examples, the controller can output an error message to the user. Additionally, in some examples, the controller can reset the cryptographic mechanism such that new public and private keys are used to generate the cryptographic signature. Other actions that may be taken when an attack is detected can include warning the driver, discarding the detected objects from the attacked object detector and relying on other secure object detectors, alerting the vehicle manufacturer about the compromised object detector, relying on input from connected infrastructure, not passing the detected object detector to other vehicle control modules, or other countermeasure / response actions. Method 900 can then end.

[0072] If method 900 determines in step 904 that the signature verification was successful, then method 900 proceeds to step 910 and includes determining the localization output. For example, the controller determines that no unauthorized modification occurred, and the localization output coordinates [X, Y, W, H, P] can be used to determine the location of the bounding box, such as the bounding box marking the location of the detected object.

[0073] At step 912, method 900 includes taking the encrypted and cryptographically signed classification output as Figure 7 the output of an object detection algorithm. For example, the cryptographically signed classification output at 912 can be Figure 7 the cryptographically signed classification output 740, and can be the cryptographically signed and encrypted output that describes the class C of the object.

[0074] At step 914, method 900 includes verifying the encrypted classification output via a public key (e.g., signature verification) (shown as "public key 2" in Figure 9 . For example, using the second public key stored in the WPM, the cryptographic signature can be verified to determine whether the cryptographic signature is valid.

[0075] At step 916, method 900 includes determining whether the signature verification at step 914 is successful. For example, if the controller determines at step 914 that the cryptographic signature is valid, then the signature verification is successful. Additionally, if the controller determines at step 914 that the cryptographic signature is invalid, then the signature verification is not successful.

[0076] If method 900 determines at step 916 that the signature verification is not successful, then method 900 proceeds to step 918 and includes determining that an object class attack may have occurred. Specifically, if the controller determines that the signature verification is not successful, then an unauthorized modification to the classification output may have occurred. For example, an external entity may have adjusted the classification output. The controller can determine not to use the classification output to adjust vehicle operation. Method 900 may then end.

[0077] At step 920, method 900 includes decrypting class C with a private key (shown as "private key 3" in Figure 9 . For example, using the private key, class C can be decrypted by the controller. The private key can be stored in the RPMB and can be provided by the vehicle manufacturer.

[0078] At step 922, method 900 determines whether the decryption of class C is successful. For example, an unsuccessfully decrypted class C can indicate that an unauthorized modification has occurred, while a successfully decrypted class C can indicate that no unauthorized modification has occurred.

[0079] If method 900 determines at step 922 that the decryption of class C is not successful, then method 900 proceeds to step 918 and includes determining that an object class attack may have occurred. For example, if the decryption of class C is not successful, then the controller determines that an unauthorized modification has occurred, and thus an object class attack has occurred. Method 900 may then end.

[0080] If method 900 determines at step 922 that the decryption of class C is successful, method 900 proceeds to step 924 and includes comparing the decryption identifier (ID) of class C with a locally stored set of IDs. For example, class C is an ID that associates a detected object with a class, such as a vehicle class, a pedestrian class, a traffic signal class, etc. Thus, the decrypted class ID is compared with the stored class IDs that were previously assigned to each object class during the encryption process. In this way, additional insertions of random class information can be detected, such as unauthorized modifications of object classes via the insertion of additional class ID information.

[0081] At step 926, method 900 includes determining an unencrypted classification output. For example, after decrypting the classification output, the controller can obtain the classification output (e.g., class C).

[0082] At step 928, method 900 includes passing the true and verified object to the vehicle control system. For example, after successfully verifying each of the localization output and the classification output, the controller can obtain the bounding box position (X, Y), the bounding box size (W, H), the object probability (P), and the object class (C). The bounding box coordinates [X, Y, W, H, P, C] can be used to adjust vehicle control. As an example, the bounding box coordinates can be used during autonomous vehicle operation to adjust vehicle operation based on detected objects. As another example, the bounding box coordinates can be used by a driver assistance system (such as a collision avoidance system) to adjust vehicle operation based on detected objects. Both the true and verified localization output 910 and the classification output 926 are passed to 928 (shown in Figure 9 as "Pass the true and verified object to autonomous driving perception"). Method 900 can then end.

[0083] In this way, the incidence of unauthorized modification of the output of the object detection system of a vehicle can be reduced. The object detection system can include a neural network that takes sensor data (e.g., such as a camera image) as input and outputs the bounding box coordinates of the detected objects and the corresponding classified object classes. The bounding box coordinates and object classes can be used by one or more vehicle systems to adjust vehicle operations, such as one or both of an autonomous vehicle control system and a driver assistance system. For example, by providing a cryptographic mechanism in the fully connected layer of the neural network of the object detection system, the output of the neural network can be cryptographically signed. For example, the localization output and the classification output can be cryptographically signed. The cryptographic signature can be verified such that unauthorized modification of the bounding box coordinates and object classes can be detected. For example, if the controller of the vehicle determines that an unauthorized modification has occurred, the controller may not use the output of the neural network to adjust vehicle operations. By enabling the controller to identify unauthorized modifications, the modified bounding boxes and object classes may not be used to adjust vehicle operations. Overall, the accuracy of the vehicle control system, the functional safety of the vehicle, and customer satisfaction can be improved.

[0084] The technical effect of including a cryptographic mechanism in the object detection system of a vehicle is that the output of the object detection algorithm can be cryptographically signed, and the cryptographic signature can be verified by the controller of the vehicle.

[0085] The description of the embodiments has been presented for purposes of illustration and description. Suitable modifications and changes to the embodiments can be made in light of the above description, or such suitable modifications and changes can be obtained through practice. For example, unless otherwise indicated, one or more of the described methods can be performed by a suitable device and / or combination of devices, such as the telematics unit 30 described with reference Figure 1 The methods can be performed by executing stored instructions using a combination of one or more logic devices (e.g., processors) and one or more additional hardware elements, such as a storage device, a memory, a hardware network interface / antenna, a switch, an actuator, a clock circuit, etc. The described methods and associated actions can also be performed in various orders other than the order described in this application, in parallel, and / or simultaneously. The described systems are exemplary in nature and can include additional elements and / or omit elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations and other features, functions, and / or properties disclosed.

[0086] As used in this application, an element or step recited in the singular and preceded with the word "a" or "an" is to be understood as not excluding a plurality of such elements or steps, unless such exclusion is stated. Further, a reference to "one embodiment" or "an example" of the present disclosure is not to be construed as excluding the existence of additional embodiments that also incorporate the recited features. The terms "first," "second," and "third," etc. are used merely as labels, and are not intended to impose numerical requirements or a particular positional order on their objects. The appended claims particularly point out the subject matter regarded as novel and non-obvious from the foregoing disclosure.

Claims

1. A method for object detection in an autonomous vehicle, comprising: Generating a cryptographic signature for the output of nodes of a fully connected layer of a neural network of a vehicle object detection system, the nodes of the fully connected layer including localization nodes and classification nodes, the cryptographic signature being based at least in part on a first private key stored in a replay protected memory block (RPMB), the output at least partially describing a detected object; And Responsive to verifying the cryptographic signature, adjusting vehicle operation based on the detected object, wherein The output that at least partially describes the detected object includes a localization output from the localization nodes and a classification output from the classification nodes, the localization output including coordinates for both the bounding box position of the detected object and the size of the bounding box of the detected object, the classification output including the object class of the detected object, and Generating the cryptographic signature for the output includes generating a first cryptographic signature for the localization output within the function of the localization nodes, and generating a second cryptographic signature for the classification output within the function of the classification nodes, the first cryptographic signature being generated based on the first private key stored in the replay protected memory block (RPMB), and the second cryptographic signature being generated based on a second private key stored in the replay protected memory block.

2. The method of claim 1, wherein verifying the cryptographic signature includes verifying the first cryptographic signature based on a first public key and verifying the second cryptographic signature based on a second public key, each of the first public key and the second public key being stored in a write protected memory (WPM) of the vehicle.

3. The method according to claim 2, further comprising: Responsive to not verifying the cryptographic signature, not adjusting the vehicle operation based on the detected object; And Outputting an error message to a user of the vehicle, or discarding the detected object from an object detector, or warning a vehicle manufacturer, or not passing the detected object to another vehicle control module.

4. The method of claim 1, wherein the localization output is based on the output of a first plurality of nodes of the fully connected layer, and the classification output is based on the output of a second plurality of nodes of the fully connected layer.

5. The method of claim 1, further comprising: Providing encryption within the classification nodes based on a third public key before generating the second cryptographic signature, the third public key being stored in a write protected memory (WPM).

6. The method of claim 1, wherein the neural network is a You Only Look Once (YOLO) neural network.

7. The method of claim 1, wherein the neural network is one of a Spatial Pyramid Pooling (SPP) neural network, a Fast R-CNN (FRCN) neural network, a Region Proposal Network (RPN) neural network, a Single Shot Detector (SSD), a Deconvolutional Single Shot Detector (DSSD), and a RetinaNet, a deformable convolutional network.

8. A method for object detection in an autonomous vehicle, comprising: Input a camera image into a neural network of an object detection system of the vehicle at the vehicle, the neural network including a fully connected layer; Within the function of a localization node including the fully connected layer, sign the localization node output with a first cryptographic signature; Within the function of a classification node including the fully connected layer, sign the classification node output with a second cryptographic signature; Process each of the signed localization node output and the signed classification node output via an activation function, the output of the activation function including a localization output signed with the first cryptographic signature and a classification output signed with the second cryptographic signature, each of the localization output and the classification output at least partially describing a detected object; Verify each of the first cryptographic signature and the second cryptographic signature based on at least one public key and at least one private key; And In response to successfully verifying each of the first cryptographic signature and the second cryptographic signature, adjust at least one vehicle control based on each of the localization output and the classification output, wherein The localization output includes coordinates for both a bounding box position of the detected object and a size of the bounding box of the detected object, the classification output includes an object class of the detected object, and Generating the cryptographic signature for the output includes generating a first cryptographic signature for the localization output within the function of the localization node, and generating a second cryptographic signature for the classification output within the function of the classification node, the first cryptographic signature being generated based on a first private key stored in a replay protected memory block (RPMB), and the second cryptographic signature being generated based on a second private key stored in the replay protected memory block.

9. The method according to claim 8, further comprising: In response to unsuccessfully verifying each of the first cryptographic signature and the second cryptographic signature, do not adjust the at least one vehicle control based on each of the localization output and the classification output, and output an error message to a user of the vehicle and / or discard the detected object from the object detector and / or alert a vehicle manufacturer and / or do not pass the detected object to another vehicle control module.

10. The method according to claim 8, wherein the first cryptographic signature is generated based on a first private key, and the second cryptographic signature is generated based on a second private key, each of the first private key and the second private key being stored in a replay protected memory block (RPMB) of the vehicle.

11. The method according to claim 8, wherein before signing the classification node output with the second cryptographic signature, encrypt the classification node output based on a first public key stored in a write protected memory (WPM) of the vehicle.

12. The method according to claim 8, wherein the classification output provides a class of the detected object, the class including one of a pedestrian class, a vehicle class, a traffic signal class, and a sign class.

13. The method according to claim 8, wherein the localization output provides coordinates of the bounding box position, the size of the bounding box, and the probability of the bounding box.

14. A system for object detection in an autonomous vehicle, comprising: a vehicle system; a vehicle control system including at least one of an autonomous vehicle control system and a driver assistance system; a plurality of sensors communicatively coupled to the vehicle control system, the plurality of sensors including at least one camera; an object detection system including a neural network that takes as input a camera image from the at least one camera and outputs coordinates corresponding to a bounding box of a detected object as an output; a controller that stores executable instructions in a non-transitory memory, the executable instructions when executed cause the controller to: generate a cryptographic signature for an output of a node of a fully connected layer of a neural network of an object detection system of the vehicle, the nodes of the fully connected layer including localization nodes configured to output a localized object bounding box of the neural network and classification nodes configured to output a class of a localized object, the cryptographic signature being at least partially based on a first private key stored in a replay protected memory block (RPMB), the output at least partially describing the detected object, and responsive to verifying the cryptographic signature, adjust vehicle operation based on the detected object, wherein the output that at least partially describes the detected object includes a localization output from the localization nodes and a classification output from the classification nodes, the localization output includes coordinates for both a bounding box position of the detected object and a size of the bounding box of the detected object, the classification output includes an object class of the detected object, and generating the cryptographic signature for the output includes generating a first cryptographic signature for the localization output within the function of the localization nodes, and generating a second cryptographic signature for the classification output within the function of the classification nodes, the first cryptographic signature is generated based on the first private key stored in the replay protected memory block (RPMB), and the second cryptographic signature is generated based on a second private key stored in the replay protected memory block.

15. The system according to claim 14, wherein the controller includes additional executable instructions stored in the non-transitory memory, the additional executable instructions when executed cause the controller to: responsive to not verifying the cryptographic signature of the output of the neural network, not adjust the autonomous vehicle control system and the driver assistance system based on the output; and output an error message indicating an unauthorized modification to the object detection system.

16. The system according to claim 14, wherein in addition to images from the at least one camera, the neural network also takes as input data from a plurality of additional sensors, the plurality of additional sensors including a GPS sensor, a lidar sensor, a radar sensor, and an ultrasonic sensor.

17. The system according to claim 14, wherein the neural network is a YOLO neural network, and the YOLO neural network includes a plurality of convolutional layers, a first fully-connected layer, a second fully-connected layer, a weight matrix, and a leaky rectified linear unit (ReLU) activation function.

Images