Processing sensor information for object detection

Abstract

A method of processing vehicle sensor information for object detection may include: capturing sensor information; generating a feature map based on the captured sensor information; associating a prior box having a set of two or more width priors and a set of two or more height priors with each pixel of the feature map; determining a confidence value for each height prior and each width prior; outputting an indication of a detected object based on the highest confidence height prior and the highest confidence width prior; and performing a vehicle operation based on the output indication of the detected object. Embodiments may include: determining one or more prior boxes having a center value, a size value, and a set of orientation priors for each pixel of the feature map; determining a confidence value for each orientation prior; and outputting an indication of the orientation of the detected object based on the highest confidence orientation.

Description

[0001] Related Applications

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 773,710, filed on November 30, 2018, entitled “Processing Sensor Information for Object Detection,” the entire contents of which are incorporated herein by reference and for all purposes.

[0003] background

[0004] Typical autonomous and semi-autonomous vehicles can detect information about their location and surrounding environment (e.g., using cameras, radar, lidar, GPS, odometry, accelerometers, and other sensors), and include control systems that interpret the sensor information to identify hazards and determine a navigation path to follow. Such vehicles include control systems to operate with limited or no control from the occupants or other operators of the vehicle. The safe operation of these vehicles depends on early and accurate identification of obstacles and tracking and predicting the movements of other vehicles in the dynamic and busy environment of streets and highways.

[0005] Some autonomous and semi-autonomous vehicles include computer systems that implement neural networks for object detection within images. Autonomous and semi-autonomous vehicle systems may use object detection systems to make driving decisions.

[0006] Overview

[0007] Various aspects include methods for improving processing of vehicle sensor information to detect objects. Various aspects include methods that can be implemented in a processor of a vehicle control unit, such as a processor within a vehicle navigation system for an autonomous or semi-autonomous vehicle or within a vehicle safety system.

[0008] Various aspects may include: capturing, by a sensor, sensor information about an environment surrounding the sensor; generating, by a processor, a feature map based on the sensor information; associating, by the processor, a prior box comprising a set of two or more width priors and a set of two or more height priors with each pixel of the feature map; determining, by the processor, a confidence value for each height prior and each width prior; outputting, by the processor, an indication of a detected object based on the highest confidence height prior from the set of height priors and the highest confidence width prior from the set of width priors; and performing, by the processor, a vehicle operation based on the output indication of the detected object.

[0009] Some aspects may include selecting a highest confidence height prior from a set of two or more height priors, and selecting a highest confidence width prior from a set of two or more width priors. Some aspects may include determining a height refinement value for each in the set of height priors, and determining a width refinement value for each in the set of width priors.

[0010] Some aspects may include refining a highest confidence height prior using a height refinement value, and refining a highest confidence width prior using a width refinement value. In some aspects, outputting an indication of a detected object based on a highest confidence height prior from a set of height priors and a highest confidence width prior from a set of width priors may include outputting a refined highest confidence height prior and a refined highest confidence width prior.

[0011] In some aspects, each associated a priori box may include a center value. Such aspects may include determining a center value refinement for each a priori box, and refining the center value using the determined center value refinement.

[0012] Some aspects may include determining one or more object classifications for each prior box and determining a confidence value for each of the one or more object classifications. In some aspects, the output indication of the detected object may include one or more of a height confidence value and a width confidence value.

[0013] Various aspects may include: capturing, by a sensor, sensor information about an environment surrounding the sensor; generating, by a processor, a feature map based on the sensor information; associating, by the processor, one or more prior boxes including a center value, a size value, and a set of orientation priors with each pixel of the feature map; determining a confidence value for each orientation prior; outputting an indication of a detected object based on the highest confidence orientation; and performing a vehicle operation based on the output indication of the detected object.

[0014] Some aspects may include selecting the highest confidence orientation prior from the orientation prior set for each prior box. Some aspects may include determining an orientation refinement for each in the orientation prior set. Some aspects may include refining the orientation value of each prior box based on the determined orientation refinement value. Some aspects may include rotating each prior box based on the refined orientation value. In some aspects, the output indication of the detected object may include an orientation confidence value.

[0015] A further aspect includes a control unit for use in a vehicle having one or more sensors, the control unit including a processor configured with processor executable instructions to perform the operations of any of the methods outlined above. A further aspect includes a vehicle including one or more sensors and a processor coupled to the one or more sensors and configured with processor executable instructions to perform the operations of any of the methods outlined above. A further aspect includes a computing device including means for performing the operations of any of the methods outlined above. A further aspect includes a non-transient processor-readable storage medium having processor executable software instructions stored thereon, the processor executable software instructions being configured to cause a processor of a computing device to perform the operations of any of the methods outlined above. A further aspect includes a system on a chip including a processor configured with processor executable instructions to perform the operations of any of the methods outlined above. A further aspect includes a system including a sensor and a computing device having a processor configured with processor executable instructions to perform the operations of any of the methods outlined above. A further aspect includes a system comprising means for performing the functions of any of the methods outlined above. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments and, together with the general description given above and the detailed description given below, serve to explain features of the various embodiments.

[0018] Figure 1A and 1B is a block diagram illustrating components of a vehicle suitable for implementing various embodiments.

[0019] Figure 1C is a component block diagram illustrating various components of a vehicle suitable for implementing various embodiments.

[0020] Figure 2 is a block diagram illustrating components of an example system-on-chip that may be configured to collect and analyze sensor information for use in a vehicle in accordance with various embodiments.

[0021] Figure 3A and 3B is a block diagram illustrating components of a neural network architecture suitable for performing object detection based on sensor information, according to various embodiments.

[0022] Figure 4A is a process flow diagram illustrating an embodiment method of processing vehicle sensor information for object detection in accordance with various embodiments.

[0023] Figure 4B is a process flow diagram illustrating an embodiment method of processing vehicle sensor information for object detection in accordance with various embodiments.

[0024] Figure 5 is a process flow diagram illustrating an embodiment method of processing vehicle sensor information for object detection in accordance with various embodiments.

[0025] Fig. 6A and 6B is a diagram illustrating the output of a neural network that performs object detection based on sensor information according to various embodiments.

[0026] Figure 7 is a block diagram illustrating components of a neural network architecture for performing object detection based on sensor information in accordance with various embodiments.

[0027] Fig. 8A is a process flow diagram illustrating an embodiment method of processing vehicle sensor information for object detection in accordance with various embodiments.

[0028] Figure 8B is a process flow diagram illustrating an embodiment method of processing vehicle sensor information for object detection in accordance with various embodiments.

[0029] Fig.9A -9C is a diagram illustrating the output of a neural network that performs object detection based on sensor information according to various embodiments.

[0030] Detailed Description

[0031] The various aspects will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to specific examples and embodiments are for illustrative purposes and are not intended to limit the scope of the various aspects or claims.

[0032] Various embodiments include methods, vehicle computing systems implementing the methods, and processing devices for use in vehicles configured to implement the methods for processing vehicle sensor information for object detection. Various embodiments can improve the operation of autonomous and semi-autonomous vehicles by providing improved efficiency and accuracy of object detection from sensor information.

[0033] As used herein, the term "sensor information" refers to environmental information captured by one or more sensors of a vehicle, which can be processed by a processor of the vehicle to detect an object. Such sensors may include radar, lidar, cameras, and other suitable sensors. Sensor information includes unprocessed information output by such sensors, as well as processed output such as by signal processing to clarify or enhance the sensor information. Sensor information is not limited to information organized or presented for human perception, such as still images or video image sequences.

[0034] Autonomous and semi-autonomous vehicles typically include multiple sensors, including cameras, radars, and lidars, that collect information about the surrounding environment. Such collected information can enable the vehicle's processor to detect roads, identify objects to avoid, and track the location and movement of other vehicles to achieve partially or fully autonomous navigation. Even when operated by an occupant or other operator, such collected information can enable the vehicle to facilitate and / or improve the occupant's or other operator's (all) reactions.

[0035] Some autonomous vehicles and semi-autonomous vehicles include a processor that implements a neural network configured to detect objects within images collected by vehicle sensors. As used herein, the term "object detection" refers to the process of detecting an object within a sensor data set. When an object is detected, the processor may output an indication that the object has been detected (e.g., an indication or information about the object having been detected), and in some cases characterize or identify the detected object with a certain confidence level. Neural network processors have been shown to be efficient in detecting and / or identifying objects within images and similar data. A neural network is a machine learning model that uses multiple layers of operations to predict one or more outputs from one or more inputs. The output of each layer is used as the input of another layer in the network. Each layer of the neural network specifies one or more transformation operations to be performed on the input of the layer. Each layer uses the current values ​​of the parameter set of the layer to generate one or more outputs. Neural networks are typically trained to identify objects within an image by providing the neural network with a relatively large data set (sometimes referred to as "ground truth") of images including the identified objects. Once trained, the neural network can process input (e.g., an image or other sensor data) to make a determination or detection of an object, which the neural network can output as an indication that an object has been detected (e.g., within a portion of an image), and in some cases as an identification or classification of the detected object (e.g., a pedestrian or vehicle). The processor can be configured to use the output of the detection or classification of the object to make decisions that affect real-world operations. For example, autonomous and semi-autonomous vehicle systems can use the output of the neural network object detection process to make navigation, steering, and other operational decisions (e.g., swerve to avoid a detected object).

[0036] Improvements in the operation of processors implementing neural networks thus provide tangible real-world benefits, including improved autonomous vehicle navigation, improved object and collision avoidance systems for vehicle safety systems, and autonomous vehicles. Such benefits may apply to any form of vehicle, including land vehicles (e.g., autonomous cars and robots), air vehicles (e.g., drones and aircraft autopilot systems), water vehicles (e.g., autonomous surface and underwater vehicles and navigation / collision avoidance systems), and spacecraft (e.g., orbital imaging systems and planetary probes). Thus, references to "vehicles" herein are intended to encompass any form of vehicle, and descriptions of embodiments using land vehicles (e.g., autonomous cars) are for illustrative purposes and are not intended to limit the scope of the claims. In addition, various embodiments may have technical applications beyond vehicle navigation and safety systems, including image recognition systems that process images through neural network processors.

[0037] Object detection performed by the neural network on the sensor information may include determining the location and semantic category or classification of the detected object (e.g., person, vehicle, sign, building, etc.). The processor may employ one or more of a variety of machine learning and deep learning approaches to perform object detection. Deep learning approaches may be used to perform object detection without specifically defining the features of the object to be detected. Some deep learning approaches may employ convolutional neural networks.

[0038] The single shot detector (SSD) architecture is a deep learning approach that uses a convolutional neural network to process sensor information (usually once) and then determine a feature map. SSD then applies a set of default or candidate boxes ("bounding boxes" or "prior boxes") at different aspect ratios and scales to the feature map. By determining the feature map as the sensor information passes through the neural network, features for the prior boxes can be extracted in a single step. SSD then generates a confidence value for each object category in each of the default prior boxes. SSD also determines an adjustment offset for the center value, height, and width of each of the default prior boxes to make the default prior box better fit the ground truth box. SSD can then perform operations such as non-maximum suppression to remove low scores or duplicate detections.

[0039] The typical SSD approach for object detection includes some inefficiencies and limitations. The application of multiple prior boxes of multiple sizes can produce a "combinatorial explosion" of complexity that increases exponentially, which increases the computational burden and inefficiency of the typical approach. The use of multiple prior boxes also limits the analysis of other features (such as the orientation or speed of the object) because adding additional priors to each prior box contributes to the "combinatorial explosion" of complexity and the computational burden.

[0040] As a non-limiting example, conventional automotive radar systems do not provide any information about the orientation of an object. It is difficult to extract information about the orientation of an object from the underlying radar signal using conventional signal processing as employed in such conventional radar systems.

[0041] Various embodiments include methods, vehicle control units implementing the methods, and computing devices configured to implement the methods for processing vehicle sensor information for object detection for use in a vehicle. Various embodiments may improve the operation of autonomous vehicles and semi-autonomous vehicles by providing improved object detection based on images or other sensor data. Specifically, various embodiments may improve the operation of computing devices that process sensor information to improve object detection. Various embodiments may improve the operation of processors that process sensor information to provide more efficient analysis of such sensor information. Various embodiments may improve the operation of processors that process sensor information by enabling more aspects of the sensor information to be processed to obtain a more detailed and accurate analysis of the sensor information. Various embodiments may improve the processing of sensor information by executing a neural network by a processor to generate a more detailed and accurate analysis of the sensor information.

[0042] In various embodiments, a sensor (e.g., of a vehicle) may capture sensor information about the environment surrounding the sensor. For example, the sensor information may include light information from a camera, which may include information about light having a detected intensity at a detected angle or orientation. As another example, the sensor information may include radar or lidar information, which may include a detected signal with distance, orientation, and intensity. In some embodiments, a processor may generate a "bird's eye view" of its environment based on such information. Various embodiments may use other suitable sensor information without limitation.

[0043] In some embodiments, a processor within a vehicle control unit may generate a feature map based on the captured sensor information. In some embodiments, the feature map may include multiple features, aspects, values, or elements that the processor may analyze. In some embodiments, the feature map may include multiple pixels. In some embodiments, each pixel of the feature map may include or be associated with multiple features, aspects, values, or elements. In some embodiments, the processor may generate multiple feature maps based on the sensor information captured from the environment. In some embodiments, the feature map may have varying resolutions to provide varying levels of detail for the processor to analyze. For example, the processor may provide more accurate detection of objects of different sizes in feature maps of different resolutions. In some embodiments, the processor may quickly analyze each of the multiple feature maps, as further described below.

[0044] In some embodiments, a processor within a vehicle control unit may associate a priori frame with each pixel of a feature map, the priori frame comprising a defined center value, a height prior set, and a width prior set. The height prior set and the width prior set provide various options for the size of the prior frame for the prior frame. The height prior set typically includes two or more height values, and the width prior set typically includes two or more width values. In some embodiments, the prior frame may not include a predefined size (i.e., a predefined height and width). In some embodiments, the prior frame may include a set of one or more additional priors. Such additional priors may include the value of a continuous parameter, for example, orientation, speed, or other suitable values.

[0045] In some embodiments, during the training process, a processor within a vehicle control unit may execute a neural network configured with a plurality of prior boxes, including, for example, a defined center value, a height prior set, and a width prior set. In some embodiments, the neural network may be provided with a labeled training data set including training examples. The processor executing the neural network may process the training data, thereby configuring the processor to analyze and provide various predictions about previously unseen sensor information. For example, in a real-time or real-world environment, the processor may execute the neural network to quickly process sensor information (or feature maps generated from such sensor information).

[0046] In some embodiments, a processor within the vehicle control unit may calculate one or more object classifications and confidence values ​​for each of the object classifications for each prior frame. For example, the processor may determine one or more identifications or classifications of an object (e.g., a person, a car, a sign, etc.). The processor may determine a confidence value (e.g., a percentage confidence) for each of the classifications. In some embodiments, the processor may select the object classification with the highest confidence value for each prior frame.

[0047] In some embodiments, a processor within the vehicle control unit may determine a center value refinement for each a priori box. In some embodiments, the processor may refine the center value of each a priori box based on the determined center value refinement.

[0048] In some embodiments, a processor within the vehicle control unit may determine a confidence value for each width prior and a confidence value for each height prior. For example, the processor may determine a percentage confidence value for each width prior and each height prior of the prior box. In some embodiments, the processor may determine a confidence value for each additional prior. The confidence value for each additional prior may include, for example, a percentage confidence value.

[0049] In some embodiments, a processor within a vehicle control unit may determine a height refinement value for each of the height prior sets. In some embodiments, a processor may determine a width refinement value for each of the width prior sets based on a feature map. In some embodiments, a processor may determine a width refinement value for each of the width prior sets based on one or more feature maps. For example, a processor may determine a width refinement value based on one or more aspects of an object represented in a feature map. In some embodiments, a processor may determine a refinement value for each additional prior of each additional prior set. In some embodiments, a processor may determine a height refinement value and a width refinement value based on a feature map.

[0050] In some embodiments, the processor in the vehicle control unit may select the highest confidence height prior from the height prior set for each prior box. In some embodiments, the processor may select the highest confidence width prior from the width prior set for each prior box. In some embodiments, the processor may select the highest confidence value of each additional prior from each additional prior set for each prior box. In some embodiments, the processor may refine the highest confidence height prior using the determined height refinement value. In some embodiments, the processor may refine the highest confidence width prior using the determined width refinement value. In some embodiments, the processor may refine the highest confidence additional prior using the determined refinement value of each additional prior.

[0051] In some embodiments, a processor within a vehicle control unit may output a proposed detection for each pixel of the feature map, the proposed detection including a refined center value, a refined highest confidence height, a refined highest confidence width, and a highest confidence object classification. In some embodiments, the proposed detection may also include one or more refined additional priors.

[0052] In some embodiments, since the processor associates the prior box with each pixel of the feature map, the processor may generate multiple corresponding proposed detections for a single object. In some embodiments, the processor may perform operations such as non-maximum suppression for suppressing multiple detections. In some embodiments, the processor may calculate the overlap of the highest confidence proposed detection with other proposed detections, and may remove or ignore lower confidence proposed detections that overlap too much (e.g., greater than a threshold overlap) with the highest confidence proposed detection, because they are likely to be detections of the same object. In some embodiments, the processor may regard the highest confidence proposed detection as the most reliable or accurate proposed detection. The processor may repeat this process until a point in time when the highest confidence is lower than a threshold confidence value in the remaining proposed detections is reached. In some embodiments, proposed detections output for many pixels of the feature map may include indications of detected objects with sufficiently low confidence so that the indicated object may not be present in the proposed detection. In some embodiments, the processor may select one or more proposed detections with an overall confidence higher than a threshold confidence. In some embodiments, the processor may remove or ignore those proposed detections with an overall confidence lower than a confidence threshold. In some embodiments, this operation may result in the elimination of many or most proposed detections.

[0053] In some embodiments, a processor within a vehicle control unit may output a final detection indication (i.e., a final indication of a detected object). In various embodiments, the processor may provide multiple final indications of detected objects for each frame of sensor information received by the processor (e.g., detecting multiple objects). In some embodiments, as mentioned above, the processor may generate multiple feature maps based on the sensor information. In some embodiments, the feature maps may have varying resolutions to provide varying levels of detail for analysis by the processor. For example, the processor may provide more accurate detection of objects of different sizes in feature maps of different resolutions. In some embodiments, the processor may quickly analyze each of the multiple feature maps. The final detection may be based on analysis of multiple feature maps using the operations described above.

[0054] In some embodiments, a processor within a vehicle control unit may perform a vehicle operation based on the output indication of the detected object (i.e., the final detection). For example, the processor may plan or execute a vehicle maneuver or operation based on the final detection. As another example, the processor may generate a command or signal to a vehicle controller to cause the vehicle to turn, speed up or slow down to avoid an obstacle, follow another vehicle, etc.

[0055] In various embodiments, the sensor may capture sensor information about the environment around the sensor, and may generate a feature map based on the sensor information. In some embodiments, the operations of capturing sensor information and generating feature maps may be similar to such operations as described above.

[0056] In some embodiments, a processor within a vehicle control unit may associate one or more prior boxes with each pixel of the feature map, the prior box including a center value, a size value (e.g., height and width), and an orientation prior set. The orientation prior set may include two or more orientation values.

[0057] In some embodiments, the processor within the vehicle control unit may calculate one or more object classifications and confidence values ​​for each of the object classifications for each prior frame. In some embodiments, the processor may select the object classification with the highest confidence value for each prior frame.

[0058] In some embodiments, a processor within the vehicle control unit may determine a center value refinement for each a priori box. In some embodiments, the processor may refine the center value of each a priori box based on the determined center value refinement.

[0059] In some embodiments, a processor within the vehicle control unit may determine an orientation refinement for each orientation prior of the set of orientation priors. In some embodiments, the processor may determine the orientation refinement based on a feature map.

[0060] In some embodiments, a processor within the vehicle control unit may select the highest confidence orientation prior from the set of orientation priors for each prior frame. In some embodiments, the processor may refine the highest confidence orientation prior using the determined orientation refinement value. In some embodiments, the processor may rotate each prior frame based on the refined orientation value.

[0061] In some embodiments, a processor within a vehicle control unit may output one or more proposed detections that include a refined center value, a refined size, a highest confidence orientation, and a highest confidence object classification.

[0062] In some embodiments, a processor within a vehicle control unit may perform operations substantially as described above for reducing or eliminating duplicate detections. As mentioned above, since the processor associates one or more prior boxes with each pixel of a feature map, the processor may generate multiple corresponding proposed detections for a single object. The processor may perform operations for suppressing multiple detections, such as non-maximum suppression. In some embodiments, the processor may calculate the overlap of the highest confidence proposed detection with other proposed detections, and the processor may remove or ignore lower confidence proposed detections that overlap too much (e.g., greater than a threshold overlap) with the highest confidence proposed detection, because they are likely to be detections of the same object. In some embodiments, the processor may consider the highest confidence proposed detection as the most reliable or accurate proposed detection. The processor repeats this until a point in time when the highest confidence is lower than a threshold confidence value in the remaining proposed detections is reached. In some embodiments, proposed detections output for many pixels of a feature map may include indications of detected objects of sufficiently low confidence so as to indicate that there may be no object present in the proposed detection. In some embodiments, the processor may select one or more proposed detections with an overall confidence higher than a threshold confidence. In some embodiments, the processor may remove or ignore those proposed detections that have an overall confidence below a confidence threshold. In some embodiments, this operation may result in the elimination of many or most proposed detections.

[0063] In other words, since the network always outputs a set number of detection proposals, it will happen that a given single object has multiple corresponding detection proposals. To eliminate these duplications, the processor may perform additional steps, including well-known algorithms such as non-maximum suppression.

[0064] In some embodiments, a processor within a vehicle control unit may output a final detection (i.e., a final indication of a detected object). In various embodiments, the processor may provide multiple final indications of detected objects for each frame of sensor information received by the processor (e.g., multiple objects may be detected). In some embodiments, as mentioned above, the processor may generate multiple feature maps based on the sensor information. The feature maps may have varying resolutions to provide varying levels of detail for analysis by the processor. For example, the processor may provide more accurate detection of objects of different sizes in feature maps of different resolutions. In some embodiments, the processor may quickly analyze each of the multiple feature maps. In some embodiments, the final detection may be based on analysis of multiple feature maps using the operations described above.

[0065] In some embodiments, a processor within a vehicle control unit may perform a vehicle operation based on the output indication of the detected object (i.e., the final detection). For example, the processor may plan or perform a vehicle maneuver or operation, such as turning, accelerating, or decelerating, based on the final detection. Since the vehicle sensors may output sensor information at a high frame rate (e.g., up to 50 frames per second or higher), the processor may use the output indication of the detected object over several sensor information frames to determine the appropriate vehicle maneuver or operation. For example, the output indications of the detected object from several sensor information frames may be combined to further increase the confidence in the detection of the object, after which the processor takes action based on the combination of the output indications of the detected object.

[0066] Various embodiments improve the operation of computing devices and processors configured to perform object detection, and improve the operation of autonomous vehicles and semi-autonomous vehicles including such computing devices and / or processors. Providing a priori sets for a priori boxes enables more efficient analysis of sensor information in a linearly scaled manner, and thus avoids the combinatorial explosion of complexity. Various embodiments enable the processor to analyze more features of sensor information, thereby providing more accurate object detection. Various embodiments provide an extensible method for processing sensor information by allowing the addition of (all) prior sets (e.g., orientation, speed, and other suitable features) for almost any attribute to the priori box while avoiding combinatorial explosion during analysis. By adding priors to the priori box (e.g., enabling the use of more number of shape categories or prior shapes), various embodiments transfer determination to the classifier and reduce refinement determination.

[0067] Various embodiments may use the training signal (e.g., training data) provided to the neural network more efficiently. For example, in conventional SSD, only the matched prior boxes are learned from the training signal, and no learning is propagated to other prior boxes. For example, if the training image includes an object that is 5 units wide and 2 units long (i.e., 5x2 shape) in the image, then in conventional SSD, only the matched prior boxes will learn from the example object. Although there may be multiple matching boxes - for example, 5x2, 5x3, 4x2, and other similar closely matched boxes - a 5x10 prior box will not learn any information about how a 5-unit wide object looks, even if the 5x10 prior box is also 5 units wide. In contrast, in various embodiments, all prior boxes are learned (i.e., trained) from the training signal / data.

[0068] Various embodiments may be implemented in various vehicles, with the example vehicle 100 being Figure 1A and 1B Reference Figure 1A and 1B, the vehicle 100 may include a plurality of sensors 102-138 disposed in or on the vehicle, which are used for various purposes involved in autonomous and semi-autonomous navigation and sensor data about objects and people in or on the vehicle 100. The sensors 102-138 may include one or more of a variety of sensors capable of detecting various information useful for environmental detection, manipulation, navigation, collision avoidance, and other vehicle operations. Each of the sensors 102-138 may communicate wired or wirelessly with the control unit 140 and with each other. Specifically, the sensors may include one or more cameras 122, 136 or other optical sensors or photoelectric sensors. The sensors may further include other types of object detection and ranging sensors, such as radar 132, lidar 138, IR sensors, and ultrasonic sensors. The sensors may further include: tire pressure sensors 114, 120, humidity sensors, temperature sensors, satellite geolocation sensors 108, accelerometers, vibration sensors, gyroscopes, gravity meters, impact sensors 130, force gauges, strain gauges, strain sensors, fluid sensors, chemical sensors, gas content analyzers, pH sensors, radiation sensors, Geiger counters, neutron detectors, biomaterial sensors, microphones 124, 134, occupancy sensors 112, 116, 118, 126, 128, proximity sensors and other sensors.

[0069] The vehicle control unit 140 may be configured with processor-executable instructions to perform various embodiments using information received from various sensors (particularly cameras 122, 136, radar 132, and / or lidar 138). In some embodiments, the control unit 140 may supplement the processing of camera images using distances and relative positions (e.g., relative azimuths) available from the radar 132 and / or lidar 138 sensors. The control unit 140 may further be configured to control the steering, braking, and speed of the vehicle 100 when operating in an autonomous mode or semi-autonomous mode using information about other vehicles determined using various embodiments.

[0070] Figure 1C is a component block diagram of a system 150 illustrating various components and supporting systems suitable for implementing various embodiments. Figure 1A , 1B 1C, the vehicle 100 may include a control unit 140, which may include various circuits and devices for controlling the operation of the vehicle 100. The control unit 140 may be coupled to and configured to control a driving control component 154, a navigation component 156, a radar / lidar component 160, a camera component 162, and one or more other sensors 158 of the vehicle 100.

[0071] As used herein, the terms "component", "system", "unit" and similar terms include computer-related entities, such as but not limited to hardware, firmware, a combination of hardware and software, software, or software in execution that is configured to perform a specific operation or function. For example, a component can be but not limited to a process, a processor, an object, an executable, a thread of execution, a program, and / or a computer running on a processor. As an illustration, both an application running on a communication device and a communication device can be referred to as a component. One or more components may reside in a process and / or a thread of execution, and a component may be localized on a processor or core and / or distributed between two or more processors or cores. In addition, these components may be executed from various non-transient computer-readable media having various instructions and / or data structures stored thereon. Each component may communicate through local and / or remote processes, function or procedure calls, electronic signals, data packets, memory read / writes, and other known computers, processors and / or communication methodologies associated with processes.

[0072] The control unit 140 may include a processor 164 configured with processor-executable instructions to control steering, navigation, and other operations of the vehicle 100, including the operations of the various embodiments. The processor 164 may be coupled to a memory 166. The control unit 162 may include an input module 168, an output module 170, and a radio module 172.

[0073] The radio module 172 may be configured for wireless communication. The radio module 172 may exchange signals 182 (e.g., command signals for controlling maneuvers, signals from navigation aids, etc.) with the network transceiver 180, and may provide the signals 182 to the processor 164 and / or the navigation unit 156. In some embodiments, the radio module 172 may enable the vehicle 100 to communicate with the wireless communication device 190 via a wireless communication link 192. The wireless communication link 192 may be a two-way or one-way communication link, and may use one or more communication protocols.

[0074] Input module 168 may receive sensor data from one or more vehicle sensors 158 as well as electronic signals from other components, including driving control component 154 and navigation component 156. Output module 170 may be used to communicate with or activate various components of vehicle 100, including driving control component 154, navigation component 156, and sensor(s) 158.

[0075] The control unit 140 may be coupled to a ride control assembly 154 to control physical elements of the vehicle 100 related to the handling and navigation of the vehicle, such as the engine, motor, throttle, steering elements, flight control elements, braking or retarding elements, etc. The ride control assembly 154 may also include components for controlling other devices of the vehicle, including environmental controls (e.g., air conditioning and heating), exterior and / or interior lighting, interior and / or exterior information displays (which may include display screens or other devices for displaying information), and other similar devices.

[0076] The control unit 140 may be coupled to the navigation component 156 and may receive data from the navigation component 156 and be configured to use such data to determine the current position and orientation of the vehicle 100 and an appropriate route to the destination. In various embodiments, the navigation component 156 may include or be coupled to a GNSS receiver system (e.g., one or more global positioning system (GPS) receivers) that enables the vehicle 100 to use a global navigation satellite system (GNSS) signal to determine its current position. Alternatively or additionally, the navigation component 156 may include a radio navigation receiver for receiving navigation beacons or other signals from a radio node (such as a Wi-Fi access point, a cellular network site, a radio station, a remote computing device, other vehicles, etc.). By controlling the driving control element 154, the processor 164 can control the vehicle 100 to navigate and maneuver. The processor 164 and / or navigation component 156 may be configured to communicate with a server 184 on a network 186 (e.g., the Internet) using a wireless connection 182 to a cellular data network 180 to receive commands for controlling maneuvers, receive data useful in navigation, provide real-time position reports, and evaluate other data.

[0077] Control unit 162 may be coupled to one or more sensors 158. Sensor(s) 158 may include sensors 102-138 as described, and may be configured to provide various data to processor 164. Control unit 140 may be coupled to radar / lidar assembly 160 and / or camera assembly 162 to control the operation of radar / lidar assembly 160 and / or camera assembly 162 and to receive sensor information from radar / lidar assembly 160 and / or camera assembly 162 and to provide such sensor information to processor 164.

[0078] Although the control unit 140 is described as separate components, in some embodiments, some or all of the components (e.g., the processor 164, the memory 166, the input module 168, the output module 170, and the radio module 172) may be integrated into a single device or module (such as a system-on-chip (SOC) processing device). Such an SOC processing device may be configured for use in a vehicle and configured to perform the operations of various embodiments when installed in the vehicle (such as configured with processor-executable instructions executed in the processor 164).

[0079] Figure 2 An example system-on-chip (SOC) architecture of a processing device SOC 200 suitable for implementing various embodiments is illustrated. Figure 1A –2, the processing device SOC 200 may include several heterogeneous processors, such as a digital signal processor (DSP) 203, a modem processor 204, an image processor 206, a mobile display processor (MDP) 207, an application processor 208, and a resource and power management (RPM) processor 217. The processing device SOC 200 may also include one or more coprocessors 210 (e.g., vector coprocessors) connected to one or more of the heterogeneous processors 203, 204, 206, 207, 208, 217. Each of the processors may include one or more cores and an independent / internal clock. Each processor / core may perform operations independently of other processors / cores. For example, the processing device SoC 200 may include a processor that executes a first type of operating system (e.g., FreeBSD, LINUX, OS X, etc.) and a processor that executes a second type of operating system (e.g., Microsoft Windows). In some embodiments, the application processor 208 may be the main processor, central processing unit (CPU), microprocessor unit (MPU), arithmetic logic unit (ALU), etc. of the SOC 200. Graphics processor 206 may be a graphics processing unit (GPU).

[0080] The processing device SoC 200 may include analog circuitry and custom circuitry 214 for managing sensor data, analog-to-digital conversion, wireless data transmission, and for performing other specialized operations, such as processing encoded audio and video signals for rendering in a web browser. The processing device SoC 200 may further include system components and resources 216, such as voltage regulators, oscillators, phase-locked loops, peripheral bridges, data controllers, memory controllers, system controllers, access ports, timers, and other similar components for supporting processors and software clients (e.g., web browsers) running on the computing device.

[0081] The processing device SoC 200 may also include a dedicated camera control circuit system (CAM) 205 that includes, provides, controls and / or manages the operation of one or more of the following: cameras 122, 136 (e.g., main camera, web camera, 3D camera, etc.), video display data from camera firmware, image processing, video pre-processing, video front end (VFE), embedded JPEG, high-definition video codec, etc. The CAM 205 may be an independent processing unit and / or include an independent or internal clock.

[0082] The processing device SOC 200 also includes a dedicated ranging device control circuit system (RANGING) 209, which includes, provides, controls and / or manages the operation of one or more components (such as radar 132, lidar 138) and other suitable devices (e.g., source / emitter, detector, and scanning components, timing electronics, orientation and position determination components, etc.). RANGING 209 can be an independent processing unit and / or include an independent or internal clock.

[0083] In some embodiments, the image processor 206 may be configured with processor executable instructions and / or dedicated hardware configured to perform the image processing analysis involved in various embodiments. For example, the image processor 206 may be configured to perform operations of receiving and processing sensor information received from a camera (e.g., 122, 136) and / or from a radar or lidar system (e.g., 132, 138) via the CAM 205 for provisioning to an object detection system.

[0084] System components and resources 216, analog and custom circuitry 214, CAM 205, and / or RANGING 209 may include circuitry for interfacing with peripheral devices (such as cameras 122, 136, electronic displays, wireless communication devices, external memory chips, etc.). Processors 203, 204, 206, 207, 208 may be interconnected to one or more memory elements 212, system components and resources 216, analog and custom circuitry 214, CAM 205, RANGING 209, and RPM processor 217 via interconnect / bus module 224, which may include a reconfigurable logic gate array and / or implement a bus architecture (e.g., CoreConnect, AMBA, etc.). Communications may be provided by advanced interconnects, such as a high-performance network-on-chip (NoC).

[0085] The processing device SoC 200 may further include an input / output module (not illustrated) for communicating with resources external to the SoC, such as a clock 218 and a voltage regulator 220. The resources external to the SoC (e.g., clock 218, voltage regulator 220) may be shared by two or more of the internal SoC processors / cores (e.g., DSP 204, modem processor 206, graphics processor 208, application processor 308, etc.).

[0086] In some embodiments, the processing device SOC 200 may be included in a control unit (e.g., 140) for use within a vehicle (e.g., 100). The control unit may include a communication link for communicating with a telephone network (e.g., 180), the Internet, and / or a network server (e.g., 184), as described.

[0087] The processing device SoC 200 may also include additional hardware and / or software components suitable for collecting sensor data from sensors, including motion sensors (e.g., accelerometers and gyroscopes), user interface elements (e.g., input buttons, touch screen displays, etc.), microphone arrays, sensors for monitoring physical conditions (e.g., position, direction, motion, orientation, vibration, pressure, etc.), cameras, compasses, GPS receivers, communication circuit systems (e.g., WLAN, WiFi, etc.), and other well-known components of modern electronic devices.

[0088] Figure 3A and 3B The components of a neural network architecture 300a suitable for performing object detection based on sensor information using various embodiments are illustrated. Figure 1A-3B, the backbone neural network architecture 302 receives input 350 (such as sensor information from a camera, radar, lidar, etc.), and performs processing on the input to generate various inputs to various classification heads or classification modules, among other things. For example, the backbone architecture 302 may provide input 304 to a classification head 312, and provide input 306 to an offset regression module 314. In addition, for each feature or attribute being estimated (e.g., height prior, width prior, orientation prior, velocity prior, and other suitable priors), the backbone architecture 302 may provide input 308 to a prior classification head 316, and provide input 310 to a prior refinement head 318. The classification head 312 may output a classification 320 (which may include one or more classifications) for each prior box. The offset regression module 314 may output an offset refinement 322 for each prior box (e.g., a center refinement value that a processor may use to refine the center of the prior box, as further described below). For each feature or attribute being estimated, the prior classification head 316 may output a prior attribute classification 324 (which may include one or more classifications) for each prior box. The prior refinement head 318 may output a prior attribute refinement 326 (which may include one or more refinements). The prior attribute refinement 326 may include a refinement value for any of the priors (e.g., a height prior refinement value, a width prior refinement value, an orientation prior refinement value, a velocity prior refinement value, etc.), which the processor may use to refine the value of one or more priors, as further described below.

[0089] In various embodiments, the backbone architecture 302 may be followed by an SSD "feed-forward" convolutional network that produces a fixed-size set of bounding boxes and scores for the presence of object class instances in those boxes, followed by a non-maximum suppression step to produce the final detection. In some embodiments, the early network layers may be based on a standard architecture for high-quality image classification (truncated before any classification layers). Various embodiments may add auxiliary structures (e.g., classification head 312, offset regression module 314, prior classification head 316, and prior refinement head 318) to the SSD to produce the desired detection, as further described below.

[0090] Figure 3BAn example of a backbone neural network architecture 302 is illustrated. A neural network (e.g., backbone architecture 302) is a machine learning model that employs multiple layers of operations to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers between an input layer and an output layer. The output of each layer is used as an input to another layer in the network (e.g., the next hidden layer or the output layer). Each layer of the neural network specifies one or more transformation operations to be performed on the inputs to that layer. Some neural network layers have operations called neurons. Each neuron receives one or more inputs and generates an output received by another neural network layer. Each neuron can receive inputs from other neurons, and each neuron can provide outputs to one or more other neurons.

[0091] The neural network architecture 302 specifies the layers included in the network and their properties, and how the neurons of each layer of the network are connected, so as to specify which layers provide their outputs as inputs to which other layers and how the outputs are provided. A processor (e.g., processors 164, 203-210, 217) may perform the transformation operations of each layer, for example, implementing one or more software modules that enable the transformation operations. Thus, a layer described as performing an operation means that the processor(s) that implement the transformation operation of the layer performs the operation.

[0092] Each layer can generate one or more outputs using the current values ​​of the parameter set of that layer. Training the neural network 302 may include continuously performing a forward pass on the input, calculating gradient values, and updating the current values ​​of the parameter set of each layer. Once the neural network 302 is trained, the final parameter set can be used to make predictions in a production system.

[0093] A convolutional neural network includes a convolutional neural network layer. A convolutional neural network layer has neuron connectivity that exploits spatially local correlations in input data. To this end, a convolutional neural network layer has sparse connectivity, where neurons in one convolutional layer receive inputs from only a small subset of neurons in a previous neural network layer. The neurons from which a neuron receives its input define the receptive field of the neuron. A convolutional neural network layer may include one or more filters, which may be defined by parameters of the layer. A convolutional neural network layer may generate an output by performing a convolution of each neuron filter with the input of the layer.

[0094] In addition, each convolutional network layer may include neurons arranged in three dimensions (e.g., including depth, width, and height dimensions). The width dimension and the height dimension may correspond to two-dimensional features of the input of a layer. The depth dimension may include one or more depth sub-layers of neurons. In some embodiments, the convolutional neural network may use weight sharing so that all neurons in the depth sub-layer have the same weights to achieve translation invariance when detecting features in the input. The convolutional neural network may also include fully connected layers and other types of layers. Neurons in a fully connected layer may receive input from each neuron in the previous neural network layer.

[0095] Figure 4A A method 400a of processing vehicle sensor information for object detection by a processor is illustrated in accordance with various embodiments. Figure 1A -4A, method 400a can be implemented in a processor (e.g., 164), a processing device (e.g., 200), and / or a control unit (e.g., 140) (variously referred to as a "processor") of a vehicle (e.g., 100). The order of operations performed in blocks 402-438 is merely illustrative, and the operations of blocks 402-438 may be performed in any order and partially simultaneously in various embodiments.

[0096] As further described below, in block 402, the processor may capture sensor information about the environment surrounding the sensor via a sensor (e.g., a camera, radar, lidar, etc.). For example, the sensor information may include an image captured by a camera or light information, which may include information about light having a detected intensity at a detected angle or orientation. As another example, the captured sensor information may include radar or lidar information, which may include a detected signal having a distance, orientation, and intensity.

[0097] In block 404, the processor may generate a feature map based on the sensor information. In some embodiments, the processor may generate a feature map to include multiple features, aspects, values, or elements extracted from the sensor data captured in block 402 in a format that the processor can analyze. In some embodiments, each pixel of the feature map may include or be associated with the multiple features, aspects, values, or elements. In some embodiments, the processor may generate multiple feature maps based on the sensor information captured from the environment in block 402. In some embodiments, the processor may generate feature maps with varying resolutions to provide varying levels of detail for the processor to analyze. For example, the processor may provide more accurate detection of objects of different sizes in feature maps of different resolutions.

[0098] In box 406, the processor may associate a priori box including a center value, a set of two or more width priors, and a set of two or more height priors with each pixel of the feature map. The height prior set and the width prior set may provide various options for the size of the prior box for the prior box. The height prior set may include two or more height values, and the width prior set may typically include two or more width values. In some embodiments, the prior box may not have a predefined size (i.e., a predefined height and width). In some embodiments, the prior box may include a set of one or more additional priors. Such additional priors may include the value of a continuous parameter, for example, an orientation, a speed, or other suitable values.

[0099] In block 414, the processor may determine a confidence value for each height prior and each width prior. For example, the processor may determine a percentage confidence value for each width prior and each height prior of the prior box. In some embodiments, the processor may determine a confidence value for each additional prior. The confidence value for each additional prior may include, for example, a percentage confidence value.

[0100] In block 436, the processor may output a final indication of the detected object.In some embodiments, the output indication of the detected object may be based on the highest confidence height prior from the set of height priors and the highest confidence width prior from the set of width priors.

[0101] In block 438, the processor may take action based on the output indication of the detected object, such as performing a vehicle operation based on the output indication of the detected object in embodiments implemented in a vehicle. For example, the processor may perform an action that results in a change of direction (such as to avoid an obstacle or follow another vehicle). In many cases, the processor may take action based on the output indication of the detected object over several sensor information frames, because the operations in method 400a may be repeated many times per second (e.g., 50 times per second or more). Thus, the output indications of the detected object from several sensor information frames may be combined to further increase the confidence in the detected object, after which the processor takes action based on the combination of the output indications of the detected object.

[0102] The processor may again capture sensor information about the environment in block 402 .

[0103] Figure 4B A method 400b of processing vehicle sensor information for object detection by a processor is illustrated in accordance with various embodiments. Figures 1A-4B, method 400a may be implemented in a processor (e.g., 164), a processing device (e.g., 200), and / or a control unit (e.g., 140) (variously referred to as a "processor") of a vehicle (e.g., 100). The order of operations performed in blocks 402-438 is merely illustrative, and the operations of blocks 402-438 may be performed in any order and partially simultaneously in various embodiments.

[0104] In block 402, the processor may capture sensor information about the environment around the sensor via a sensor (e.g., a camera, a radar, a lidar, etc.), as shown in FIG. Figure 4A As described with respect to like numbered blocks in method 400a.

[0105] In block 404, the processor may generate a feature map based on the sensor information, such as referring to Figure 4A As described with respect to like numbered blocks in method 400a.

[0106] In block 406, the processor may associate a prior box including a center value, a set of two or more width priors, and a set of two or more height priors with each pixel of the feature map, as described with reference to Figure 4A As described with respect to like numbered blocks in method 400a.

[0107] In block 408, the processor may calculate one or more object classifications and confidence values ​​for each of the one or more object classifications for each prior box. For example, the processor may determine one or more identifications of an object (e.g., a person, a car, a sign, etc.). The processor may determine a confidence value (e.g., a percentage confidence) for each of the classifications.

[0108] In block 410 , the processor may select the object classification having the highest confidence value for each prior box.

[0109] In block 412, the processor may determine a center value refinement for each a priori box. In some embodiments, the processor may refine the center value of each a priori box based on the determined center value refinement.

[0110] In block 414, the processor may determine a confidence value for each height prior and a confidence value for each width prior, as described with reference to Figure 4A As described with respect to like numbered blocks in method 400a.

[0111] In block 416 , the processor may determine a height refinement value for each of the height prior sets.

[0112] In block 418 , the processor may determine a width refinement value for each of the width prior sets. In some embodiments, the processor may determine the height refinement value and the width refinement value based on the feature map generated in block 404 .

[0113] In block 420 , the processor may select, for each prior box, a highest confidence height prior from among the set of height priors.

[0114] In block 422 , the processor may select, for each prior box, a highest confidence width prior from among a set of width priors.

[0115] In block 424 , the processor may refine the center value of each prior box based on the determined center value refinement.

[0116] In block 426 , the processor may utilize the determined height refinement value to refine the highest confidence height prior.

[0117] In block 428 , the processor may utilize the determined width refinement value to refine the highest confidence width prior.

[0118] In block 430 , the processor may output a proposed detection for each pixel, the proposed detection including a refined center value, a refined highest confidence height, a refined highest confidence width, and a highest confidence object classification.

[0119] In block 432, the processor may perform operations for suppressing multiple detections (eg, non-maximum suppression).

[0120] In block 434, the processor may remove proposal detections having a confidence level below a confidence threshold. In some embodiments, the processor may perform the operations of block 434 as part of or in conjunction with the operations of block 432.

[0121] In block 436, the processor may output a final indication of the detected object, such as reference Figure 4A As described for like numbered blocks in method 400a. In some embodiments, the output indication of a detected object may be based on a highest confidence height prior from among a set of height priors and a highest confidence width prior from among a set of width priors.

[0122] In block 438, the processor may perform a vehicle operation based on the output indication of the detected object, such as referring to Figure 4AAs described for like numbered blocks in method 400a. The processor may perform vehicle operations based on output indications of detected objects over several sensor information frames, as the operations in method 400b may be repeated many times per second (e.g., 50 times per second or more). Thus, output indications of detected objects from several sensor information frames may be combined to further increase confidence in object detection, before the processor takes action based on the combination of output indications of detected objects.

[0123] The processor may again capture sensor information about the environment in block 402 .

[0124] Figure 5 A method 500 of processing vehicle sensor information for object detection by a processor is illustrated in accordance with various embodiments. Figure 1A -5, the method 500 may be implemented in a processor (e.g., 164), a processing device (e.g., 200), and / or a control unit (e.g., 140) (variously referred to as a "processor") of a vehicle (e.g., 100). The order of operations performed in blocks 402-438 and 502-512 is merely illustrative, and the operations of blocks 402-438 may be performed in any order and partially simultaneously in various embodiments. In blocks 402-438, the processor may perform the operations of the same numbered blocks of methods 400a and 400b as described.

[0125] In block 502, the processor may associate a prior box including a center value, a width prior set, a height prior set, and one or more additional prior sets with each pixel of the feature map. In some embodiments, the additional priors may include values ​​of continuous parameters, such as orientation, speed, or other suitable values. In some embodiments, the prior boxes are scalable in that they can be configured to have priors of any number of values ​​or characteristics.

[0126] After performing the operations in blocks 402 and 404 of method 400a as described, the processor may determine a confidence value for each additional prior in block 504. In some embodiments, the confidence value for each additional prior may include, for example, a percentage confidence value.

[0127] In block 506 , the processor may determine a refinement value for each additional prior of each set of additional priors.

[0128] In block 508 , the processor may select, for each prior box, the highest confidence value for each additional prior from among the set of additional priors.

[0129] In block 510 , the processor may refine one or more highest confidence additional priors using the determined refinement values ​​for each prior.

[0130] In box 512, the processor may output a refined prior box for each pixel, the refined prior box including a refined center value, a refined highest confidence height, a refined highest confidence width, refined additional one or more priors, and a highest confidence object classification.

[0131] Fig. 6A illustrates an output indication of a detected object 600a from a neural network performing object detection based on sensor information in accordance with various embodiments, and Figure 6B The final indication of the detected object 600b is illustrated. Figure 1A –6B, the neural network can be implemented in a processor (e.g., 164), a processing device (e.g., 200), and / or a control unit (e.g., 140) (variously referred to as a “processor”) of a vehicle (e.g., 100).

[0132] In some embodiments, the processor may output an indication of a detected object including a center refinement or an offset refinement 602 (eg, offset detection 322). The processor may refine the center value of each prior box using the center refinement, as described above.

[0133] In some embodiments, the processor may output an indication of the detected object including a classification 604 of each prior box. For example, the classification may include one or more identifications of the object. The classification may also include a percentage confidence of each identification. For example, the classification 604 includes identifications of the object as part of the image background, vehicles, and pedestrians. The classification 604 may also include a confidence of each classification, for example, a 5% confidence that the object is the image background, an 85% confidence that the object is a vehicle, and a 10% confidence that the object is a pedestrian.

[0134] In some embodiments, the processor may output an indication of a detected object including one or more height priors 613. Each height prior 613 may include a height value (e.g., as illustrated, 3m, 4m, 5m, and 6m). The processor may output a confidence value for each height classification 606 (e.g., as illustrated, 10% confidence for 3m, 15% confidence for 4m, 54% confidence for 5m, 21% confidence for 6m). In some embodiments, the processor executing the neural network may output an indication of a detected object including a series of values, to which the processor may apply a function that transforms these values ​​into probabilities that sum to 1 (e.g., a Softmax function). The processor may also output an indication of a detected object including one or more height refinement values ​​(i.e., (s) prior height refinement) 608, which may correspond to the one or more height priors (e.g., +0.5m, +0.8m, -0.13m, and -0.4m). The processor may use the height refinement value to refine the highest confidence height classification, as further described below.

[0135] In some embodiments, the processor may output an indication of the detected object including one or more width priors 615. Each width prior 615 may include a weight value (e.g., as illustrated, 1m, 2m, and 2.5m). The processor may output a confidence value for each width classification 610 (e.g., as illustrated, 10% confidence for 1m, 70% confidence for 2m, and 20% confidence for 2.5m). In some embodiments, the processor executing the neural network may output an indication of the detected object including a series of values, to which the processor may apply a function that transforms these values ​​into probabilities that sum to 1 (e.g., a Softmax function). The processor may also output an indication of the detected object including one or more width refinement values ​​(i.e., (s) prior width refinement) 612, which may correspond to the one or more width priors (e.g., +0.2m, -0.1m, and -0.8m). The processor may use the width refinement values ​​to refine the highest confidence width classification, as further described below.

[0136] In some embodiments, the processor may select the highest confidence height prior 614 , and the highest confidence width prior 616 .

[0137] Based on the output indication 600a of the detected object, the processor may output a final detection indication or information (e.g., a final indication of the detected object) 600b. The final indication 600b of the detected object may include a category / classification 620 of the object (e.g., a vehicle), which may reflect the highest confidence classification 604 (e.g., 85% confidence). The final indication 600b of the detected object may include a refined center 622 of the priori box (e.g., 17.2m, 22.8m, which corresponds to a position within the sensor information). The refined center 622 of the priori box may be based on the center of the priori box plus or minus a center refinement (or offset refinement). For example, the processor may refine the priori box center value (17m, 23m) using the center refined value (+0.2m, -0.3m) to output a refined center value (17.2m, 22.8m). The final indication 600b of the detected object may include a refined highest confidence height prior 624 (e.g., 5.13 m). In some embodiments, the processor may refine the height prior value (e.g., 5 m) using the value of the corresponding prior height refinement value 608 (e.g., +0.13 m) to output a refined highest confidence height (e.g., 5.13 m). The final indication 600b of the detected object may include a refined highest confidence width prior 626 (e.g., 1.9 m). In some embodiments, the processor may refine the width prior value (e.g., 2 m) using the value of the corresponding prior width refinement value 612 (e.g., -0.1 m) to output a refined highest confidence width 626 (e.g., 1.9 m). The final indication 600b of the detected object may include the confidence of the highest confidence height prior 628 (e.g., 54%), the confidence of the category / classification 630 (e.g., 85%), and the confidence of the highest confidence width prior 632 (e.g., 70%).

[0138] Figure 7 Components of a neural network architecture 700 suitable for implementing various embodiments to perform object detection based on sensor information are illustrated. Figure 1A –7, backbone neural network architecture 702 (an example of which is shown in Figure 3B 702 receives input 750 (such as sensor information from a camera, radar, lidar, etc.) and performs processing on the input to generate various inputs to various classification heads or classification modules, among others. For example, the backbone architecture 702 may provide input 704 to a box regression headend 712, input 706 to a classification headend 714, input 708 to an orientation classification headend 716, and input 710 to an orientation regression headend 718.

[0139] The box regression head 712 may output a box regression 720 for each prior box. The classification head 714 may output a classification 722 for each prior box. The orientation classification head 716 may output an orientation classification 724 for each prior box. The orientation regression head 718 may output an orientation regression 726 for each prior box.

[0140] As described, the backbone architecture 302 may be followed by an SSD "feed-forward" convolutional network that produces a fixed-size set of bounding boxes and scores of the presence of object class instances in those boxes, followed by a non-maximum suppression step to produce the final detection. In some embodiments, the early network layers may be based on a standard architecture for high-quality image classification (truncated before any classification layer). Various embodiments may add auxiliary structures (e.g., box regression head 712, classification head 714, orientation classification head 716, and orientation regression head 718) to the SSD to produce the desired detection. The processor may use each of the outputs (detections) 720-726 from each of the head / modules 712-718, as further described below.

[0141] Fig. 8A A method 800a of processing vehicle sensor information for object detection by a processor is illustrated in accordance with various embodiments. Figure 1A-8A , method 800a can be implemented in a processor (e.g., 164), a processing device (e.g., 200), and / or a control unit (e.g., 140) (variously referred to as a "processor") of a vehicle (e.g., 100). The order of operations performed in blocks 802-834 is merely illustrative, and the operations of blocks 802-834 may be performed in any order and partially simultaneously in various embodiments.

[0142] In block 802 , a processor may capture sensor information about an environment surrounding the sensor via a sensor (eg, a camera, radar, lidar, etc.).

[0143] At block 804, the processor may generate a feature map based on the sensor information. The processor may generate a feature map based on the sensor information. Figure 4A The feature map is generated in an operation similar to the operation described with respect to block 404 of method 400a.

[0144] In block 806, the processor may associate one or more prior boxes including a center value, a size value, and an orientation prior set with each pixel of the feature map. In some embodiments, the orientation prior set may include two or more orientation values.

[0145] In block 808, the processor may determine a confidence value for each orientation prior.

[0146] In block 836, the processor may output a final indication of the detected object.In some embodiments, the output indication of the detected object may be based on the highest confidence orientation prior among the set of orientation priors.

[0147] In block 838, the processor may perform a vehicle operation based on the output indication of the detected object, such as generating a command to the vehicle control system to turn, slow down, or speed up the vehicle. The processor may perform a vehicle operation based on the output indication of the detected object over several sensor information frames, as the operations in method 800a may be repeated many times per second (e.g., 50 times per second or more). Thus, the output indications of the detected object from several sensor information frames may be combined to further increase confidence in the object detection, after which the processor takes action based on the combination of the output indications of the detected object.

[0148] The processor may again capture sensor information about the environment in block 802 .

[0149] Figure 8B A method 800b of processing vehicle sensor information for object detection by a processor is illustrated in accordance with various embodiments. Figures 1A-8B , method 800b may be implemented in a processor (e.g., 164), a processing device (e.g., 200), and / or a control unit (e.g., 140) (variously referred to as a "processor") of a vehicle (e.g., 100). The order of operations performed in blocks 802-834 is merely illustrative, and the operations of blocks 802-834 may be performed in any order and partially simultaneously in various embodiments.

[0150] In block 802, the processor may capture sensor information about the environment around the sensor via a sensor (e.g., a camera, a radar, a lidar, etc.), as shown in FIG. Fig. 8A As described with respect to like numbered blocks in method 800a.

[0151] In block 804, the processor may generate a feature map based on the sensor information, such as referring to Fig. 8A As described with respect to like numbered blocks in method 800a.

[0152] In block 806, the processor may associate one or more prior boxes including a center value, a size value, and an orientation prior set with each pixel of the feature map, as described with reference to Fig. 8A As described for like numbered blocks in method 800a. In some embodiments, the orientation prior set may include two or more orientation values.

[0153] In block 808, the processor may determine a confidence value for each orientation prior, such as by referring to Fig. 8A As described with respect to like numbered blocks in method 800a.

[0154] In block 810 , the processor may calculate one or more object classifications and a confidence value for each of the one or more object classifications for each a priori box.

[0155] In block 812 , the processor may select the object classification having the highest confidence value for each prior box.

[0156] In block 814 , the processor may determine a center value refinement for each prior box.

[0157] In block 816, the processor may determine a size refinement for each a priori box. In some embodiments, the size refinement may include a height refinement and a width refinement.

[0158] In block 818, the processor may determine an orientation refinement for each of the orientation prior sets.

[0159] In block 820 , the processor may select, for each prior box, a highest confidence orientation prior from among the set of orientation priors for each prior box.

[0160] In block 822 , the processor may refine the center value of each prior box based on the determined center value refinement.

[0161] In block 824 , the processor may refine the size value of each a priori box based on the determined size refinement.

[0162] In block 826 , the processor may refine the orientation value of each a priori box based on the determined orientation refinement value.

[0163] In block 828, the processor may rotate each a-priori box based on the refined orientation values.

[0164] In block 830 , the processor may output one or more proposed detections for each pixel of the feature map, the one or more proposed detections comprising a refined center value, a refined size, a highest confidence orientation, and a highest confidence object classification.

[0165] In block 832, the processor may perform operations for suppressing multiple detections, such as non-maximum suppression.

[0166] In block 834, the processor may remove proposal detections having a confidence level below a confidence threshold. In some embodiments, the processor may perform the operations of block 834 as part of the operations of block 832 or in conjunction with the operations of block 432.

[0167] In block 836, the processor may output a final indication of the detected object, such as reference Fig. 8AAs described for like numbered blocks in method 800a. In some embodiments, the output indication of a detected object may be based on a highest confidence orientation prior from among a set of orientation priors.

[0168] In block 838, the processor may perform a vehicle operation based on the output indication of the detected object, such as referring to Fig. 8A As described for like numbered blocks in method 800a. The processor may perform vehicle operations based on output indications of detected objects over several sensor information frames, because the operations in method 800b may be repeated many times per second (e.g., 50 times per second or more). Thus, output indications of detected objects from several sensor information frames may be combined to further increase confidence in the detection of the object, after which the processor takes action based on the combination of output indications of detected objects.

[0169] The processor may again capture sensor information about the environment in block 802 .

[0170] Fig.9A -9C is a diagram illustrating output indications 900a, 900b, and 900c of detected objects from a neural network performing object detection including determining object orientation based on sensor information in accordance with various embodiments. Figure 1A –9C, the neural network can be implemented in a processor (e.g., 164), a processing device (e.g., 200), and / or a control unit (e.g., 140) (variously referred to as a “processor”) of a vehicle (e.g., 100).

[0171] In some embodiments, the processor may associate each pixel of the feature map with one or more a priori boxes 902 as described. In some embodiments, the processor may output an indication of the detected object including a center value refinement or an offset refinement 920 (e.g., ±X, ±Y). The processor may refine the center value of each a priori box 902 using the center refinement, as described above.

[0172] In some embodiments, the processor may output an indication of the detected object including a size refinement 921. In some embodiments, the size refinement may include a width refinement 922 (e.g., ±W) and / or a height refinement 924 (e.g., ±H). In some embodiments, the processor may refine the size value of each prior box based on the determined size refinement 921.

[0173] In some embodiments, the processor may output an indication of the detected objects including a classification 926 of each prior box. For example, the classification may include one or more identifications of the object. The processor may also output an indication of the detected objects including a percentage confidence associated with each classification. For example, the classification 926 includes identification of the object as part of the image background, a vehicle, and a pedestrian. The processor may output an indication of the detected objects including a confidence value for each proposed identification (indicated as %C for each classification).

[0174] In some embodiments, the processor may output an indication of detected objects including one or more orientation predictions 928 for each a priori box (as illustrated in circles, north (or 0°), east (or 90°), south (or 180°), and west (or 270°), respectively. For each orientation 928, the processor may output an orientation confidence or orientation classification 930 (indicated as %C for each classification). In addition, the processor may output an indication of detected objects including an orientation refinement or regression 932 associated with each orientation. (As illustrated, each orientation refinement 932 corresponds to a curved arrow outside each of the circles illustrating the orientation 928.) In some embodiments, the processor may refine the orientation of each a priori box based on the determined orientation refinement.

[0175] In some embodiments, the processor may select a highest confidence orientation (e.g., yaw) prior 934. Based on the highest confidence orientation prior, the processor may output an indication of the detected object including a final orientation (e.g., yaw) prediction 936. In some embodiments, the processor may refine the highest confidence orientation 934 (e.g., 90°) using the value of the corresponding refinement (or regression) 932 (e.g., -8°) to output a refined highest confidence orientation 936 (e.g., 82°). In some embodiments, based on the final orientation prediction 936, the processor may rotate each prior box 904 based on the refined orientation value. The processor may output a final indication of the detected object 906 including the refined highest confidence orientation.

[0176] Fig. 9CAn output indication 900c of a detected object is illustrated as an example of an output indication 900b of a detected object including an orientation of a vehicle. In some embodiments, a processor may receive sensor information (such as a radar or lidar return signal) and generate a "bird's eye view" of an environment based on the sensor information. For example, the processor may output an indication of a detected object including one or more vehicle orientation predictions 950 for each prior box (e.g., north (or 0°), east (or 90°), south (or 180°), and west (or 270°), respectively. For each orientation 950, the processor may output an orientation confidence or orientation classification 952 (indicated as %C for each classification). In addition, the processor may output an orientation refinement or regression 954 associated with each orientation (illustrated as a curved arrow accompanying each orientation 950 and corresponding to an indication of ± degrees in each refinement 954).

[0177] In some embodiments, the processor may refine the orientation of each prior box based on the determined orientation refinement. In some embodiments, the processor may select the highest confidence orientation (e.g., yaw) prior 956. Based on the highest confidence orientation prior, the processor may output a final orientation (e.g., yaw) prediction 958. The processor may output a final indication of the detected object including the refined highest confidence orientation.

[0178] Various embodiments may improve the operation of autonomous vehicles and semi-autonomous vehicles by providing improved object detection. Specifically, various embodiments may improve the operation of a processor that processes sensor information to provide more accurate object detection. Various embodiments may improve the operation of a processor that processes sensor information to provide a more efficient analysis of such sensor information. Various embodiments may improve the operation of a processor that processes sensor information by enabling more features of the sensor information to be extracted and processed to obtain a more detailed and accurate analysis of the sensor information. Various embodiments may also be useful in other applications where object detection is performed based on sensor information. Therefore, unless explicitly stated in a claim element, the claims should not be interpreted as being limited to vehicle systems.

[0179] The various embodiments illustrated and described are provided merely as examples of the various features of the claims. However, the features shown and described with respect to any given embodiment are not necessarily limited to the associated embodiment, and may be used or combined with other embodiments shown and described. Furthermore, the claims are not intended to be limited to any one example embodiment.

[0180] The above method descriptions and process flow charts are provided only as illustrative examples and are not intended to require or imply that the blocks of the various embodiments must be executed in the order given. As will be appreciated by those skilled in the art, the order of the blocks in the aforementioned embodiments may be executed in any order. Phrases such as "thereafter," "then," "next," etc. are not intended to limit the order of the blocks; these phrases are simply used to guide the reader through the description of the method. Further, any reference to a claim element in singular form (e.g., a reference using the article "one," "some," or "the") should not be interpreted as limiting the element to the singular.

[0181] The various illustrative logic blocks, modules, circuits, and algorithm blocks described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and blocks are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Technicians may implement the described functionality in different ways for each specific application, but such embodiment decisions should not be interpreted as causing a departure from the scope of the various embodiments.

[0182] The hardware used to implement the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device designed to perform the functions described herein, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of communication devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some blocks or methods may be performed by circuit systems dedicated to a given function.

[0183] In various embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or codes on a non-transient computer-readable medium or a non-transient processor-readable medium. The operation of the method or algorithm disclosed herein may be implemented in a processor-executable software module, which may reside on a non-transient computer-readable or processor-readable storage medium. A non-transient computer-readable or processor-readable storage medium may be any storage medium that can be accessed by a computer or processor. As an example and not limitation, such non-transient computer-readable or processor-readable media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, disks and discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and blue-ray discs, wherein disks often reproduce data magnetically and discs reproduce data optically with lasers. The above combinations are also included within the scope of non-transient computer-readable and processor-readable media. In addition, the operations of a method or algorithm may reside as a code and / or instruction or any code and / or instruction combination or set on a non-transient processor-readable medium and / or computer-readable medium that can be incorporated into a computer program product.

[0184] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments of the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the embodiments. Thus, the embodiments are not intended to be limited to the embodiments shown herein, but should be granted the widest scope consistent with the following claims and the principles and novel features disclosed herein.

Claims

1. A method of processing sensor information for object detection, comprising: capturing, by a sensor, sensor information about an environment surrounding the sensor; generating, by a processor, a feature map based on the sensor information; Associating, by the processor, a prior box comprising a set of two or more width priors, a set of two or more height priors, and a set of two or more orientation priors with each pixel of the feature map; determining, by the processor, a confidence value for each height prior, each width prior, and each orientation prior; outputting, by the processor, an indication of a detected object based on a highest confidence height prior from the set of height priors, a highest confidence width prior from the set of width priors, and a highest confidence orientation prior from the set of orientation priors; and An action is taken based on the output indication of the detected object.

2. The method of claim 1, wherein: Taking action based on the output indication of the detected object includes: A vehicular operation is performed, by the processor, based on the output indication of the detected object.

3. The method of claim 1, further comprising: selecting the highest confidence height prior from the set of two or more height priors; as well as The highest confidence width prior is selected from the set of two or more width priors.

4. The method of claim 1, further comprising: determining a height refinement value for each of the set of height priors; as well as A width refinement value is determined for each of the set of width priors.

5. The method of claim 4, further comprising: Refining the highest confidence height prior using the height refined value; as well as The highest confidence width prior is refined using the width refinement value.

6. The method of claim 5, wherein: Outputting an indication of a detected object based on a highest confidence height prior from the set of height priors and a highest confidence width prior from the set of width priors comprises: The output includes the indication of the detected object including a refined highest confidence height prior and a refined highest confidence width prior.

7. The method of claim 1, wherein: The prior box associated with each pixel further includes a center value.

8. The method of claim 7, further comprising: Determine the center value refinement for each prior box; as well as The center value is refined using the determined center value refinement.

9. The method of claim 1, further comprising: Determine one or more object categories for each prior box; as well as A confidence value is determined for each of the one or more object classifications.

10. The method of claim 1, wherein: The output indication of the detected object includes one or more of a height confidence value and a width confidence value.

11. A vehicle control unit, comprising: A processor configured with processor-executable instructions for performing operations comprising: receiving sensor information from a sensor; generating a feature map based on the sensor information; Associating a prior box including a set of two or more width priors, a set of two or more height priors, and a set of two or more orientation priors with each pixel of the feature map; determining a confidence value for each height prior, each width prior, and each orientation prior; outputting an indication of a detected object based on a highest confidence height prior from the set of height priors, a highest confidence width prior from the set of width priors, and a highest confidence orientation prior from the set of orientation priors; and An action is taken based on the output indication of the detected object.

12. The vehicle control unit of claim 11, wherein: The processor is configured with processor-executable instructions for performing operations such that taking an action based on the output indication of the detected object comprises: A vehicular operation is performed based on the output indication of the detected object.

13. The vehicle control unit of claim 11, wherein: The processor is configured with processor-executable instructions for performing operations, the operations further comprising: selecting the highest confidence height prior from the set of two or more height priors; and The highest confidence width prior is selected from the set of two or more width priors.

14. The vehicle control unit of claim 11, wherein: The processor is configured with processor-executable instructions for performing operations, the operations further comprising: determining a height refinement value for each of the set of height priors; and A width refinement value is determined for each of the set of width priors.

15. The vehicle control unit of claim 14, wherein: The processor is configured with processor-executable instructions for performing operations, the operations further comprising: refining the highest confidence height prior using the height refined value; and The highest confidence width prior is refined using the width refinement value.

16. The vehicle control unit of claim 15, wherein: The processor is configured with processor-executable instructions for performing operations such that outputting an indication of a detected object based on a highest confidence height prior from among the set of height priors and a highest confidence width prior from among the set of width priors comprises: The output includes the indication of the detected object including a refined highest confidence height prior and a refined highest confidence width prior.

17. The vehicle control unit of claim 11, wherein: The processor is configured with processor-executable instructions for performing operations such that each associated a priori box further includes a center value.

18. The vehicle control unit of claim 17, wherein: The processor is configured with processor-executable instructions for performing operations, the operations further comprising: Determine the center value refinement for each prior box; and The center value is refined using the determined center value refinement.

19. The vehicle control unit of claim 11, wherein: The processor is configured with processor-executable instructions for performing operations, the operations further comprising: Determining one or more object classifications for each prior box; and A confidence value is determined for each of the one or more object classifications.

20. The vehicle control unit of claim 11, wherein: The processor is configured with processor-executable instructions for performing operations such that the output indication of the detected object includes one or more of a height confidence value and a width confidence value.

21. A method of processing vehicle sensor information for object detection, comprising: capturing, by a sensor, sensor information about an environment surrounding the sensor; generating, by a processor, a feature map based on the sensor information; Associating, by the processor, one or more prior boxes including a center value, a size value, and an orientation prior set with each pixel of the feature map; determining, by the processor, a confidence value for each orientation prior; outputting, by the processor, an indication of the detected object based on the highest confidence orientation; as well as A vehicular operation is performed, by the processor, based on the output indication of the detected object.

22. The method of claim 21, further comprising: For each prior box, the highest confidence orientation prior is selected from the set of orientation priors.

23. The method of claim 21, further comprising: An orientation refinement is determined for each of the set of orientation priors.

24. The method of claim 23, further comprising: The orientation value of each prior box is refined based on the determined orientation refinement.

25. The method of claim 24, further comprising: Each prior box is rotated based on the refined orientation value.

26. The method of claim 21, wherein: The output indication of the detected object comprises an orientation confidence value.

27. A vehicle control unit, comprising: A processor configured with processor-executable instructions for performing operations comprising: receiving sensor information about an environment surrounding the sensor; generating, by a processor, a feature map based on the sensor information; Associating, by the processor, one or more prior boxes including a center value, a size value, and an orientation prior set with each pixel of the feature map; determining, by the processor, a confidence value for each orientation prior; outputting, by the processor, an indication of the detected object based on the highest confidence orientation; and The processor performs a vehicle operation based on the output indication of the detected object. do.

28. The vehicle control unit of claim 27, wherein: The processor is configured with processor-executable instructions for performing operations, the operations further comprising: For each prior box, the highest confidence orientation prior is selected from the set of orientation priors.

29. The vehicle control unit of claim 27, wherein: The processor is configured with processor-executable instructions for performing operations, the operations further comprising: determining an orientation refinement for each of the set of orientation priors; and The orientation value of each prior box is refined based on the determined orientation refinement.

30. The vehicle control unit of claim 29, wherein: The processor is configured with processor-executable instructions for performing operations, the operations further comprising: Each prior box is rotated based on the refined orientation value.

Images