Computer-Implemented Vehicle Perception Method

Abstract

Examples of techniques for using fixed-point quantization in deep neural networks are disclosed. In one embodiment according to the aspects of the present disclosure, a computer-implemented method includes acquiring a plurality of images from a camera associated with a vehicle and storing image data associated with the plurality of images in a memory. The method further includes allocating vehicle perception tasks to a plurality of processing elements connected to an accelerator associated with the memory.The method further includes performing the vehicle perception tasks using a neural network through at least one of the plurality of processing elements, wherein performing the vehicle perception tasks includes quantizing a fixed-point value based on an activation input and a synaptic weight. The method further includes controlling the vehicle based at least partially on a result of performing the vehicle perception tasks.

Description

[0001] The present disclosure relates to the control of a vehicle and, in particular, to fixed-point quantization in neural networks for vehicle perception systems.

[0002] Machine learning involves creating and using algorithms capable of learning from data and making predictions. Such algorithms are typically based on building a model from example inputs to make data-driven predictions or decisions. A number of approaches to machine learning have already been developed. One such approach, known as an artificial neural network (ANN) or simply a neural network (NN), is a learning algorithm inspired by the structure and function of biological neural networks.

[0003] A neural network (NN) comprises hierarchical layers of interconnected groups of artificial neurons (nodes), with each layer of nodes receiving the outputs of a lower layer as input. Deep neural networks (DNNs) are a type of NN that includes one or more hidden layers of nodes. Feedforward NNs are NNs in which connections between nodes do not form a cycle. That is, feedforward NNs are NNs in which information moves in only one direction: forward, from the input nodes, through the nodes of one or more hidden layers (if any), and finally to the output nodes. Convolutional NNs belong to a class of deep, feedforward NNs that include input and output layers as well as multiple hidden layers. The hidden layers typically include convolutional layers, pooling layers, fully connected layers, and normalization layers.Technical improvements to conventional neural networks are described herein.

[0004] US Patent 2016 / 0328646A1 discloses a method for quantizing a machine learning floating-point network to obtain a machine learning fixed-point network using a quantizer. The method may include selecting at least one moment of an input distribution of the machine learning floating-point network. The method may also include determining quantization parameters for quantizing values ​​of the machine learning floating-point network, which are based at least partially on the at least one selected moment of the input distribution of the machine learning floating-point network, in order to obtain corresponding values ​​of the machine learning fixed-point network.

[0005] CN 1 08 319 909 A discloses a method and system for driver behavior analysis. The method comprises the following steps: receiving a real-time road condition image transmitted by a vehicle-mounted device; using a CNN model to identify the real-time road condition image and detect an object type within the real-time road condition image; and using an LSTM model to perform a semantic prediction for the object type and output appropriate prompt information.

[0006] LIN, Darryl D.; TALATHI, Sachin S.; ANNAPUREDDY, V. Sreekanth: Fixed point quantization of deep convolutional networks. In: Proceedings of the 33rd International Conference on Machine Learning, 20-22 June 2016, New York, NY, USA. Red Hook, NY, USA: Curran, 2016 (Journal of machine learning research. Workshop and conference proceedings: 48), disclose an algorithm for converting a pretrained floating-point DCN model into a fixed-point model. The algorithm optimizes the bit width for each layer of the DCN to maximize the signal-to-noise ratio (SQNR) while minimizing the model size. This is achieved by analyzing the distribution of weights and activations across the DCN layers. The procedure includes capturing activations through a forward pass, collecting statistics on weights, biases and activations, and determining the optimal fixed-point formats for each layer.The bit width is optimized layer by layer to reduce the overall complexity of the model without compromising classification accuracy.

[0007] NA, Taesik et al.: On-chip training of recurrent neural networks with limited numerical precision. IJCNN 2017: The International Joint Conference on Neural Networks, 14-19 May 2017, Anchorage, AK, USA. Piscataway: IEEE, 2017. pp. 3716-3723. ISBN 978-1-5090-6182-2. DOI: 10.1109 / IJCNN.2017.7966324 reveals that the training of neural networks can be accelerated by using limited numerical precision in conjunction with specialized low-precision hardware. A gated recurrent unit (GRU) with a dynamic fixed point is used as the numerical target format. Batch normalization of input sequences can accelerate training with both low and high precision. Low-precision training with various rounding options, including bit truncation, rounding to the nearest value, and stochastic rounding, is revealed.

[0008] The goal can be seen as providing a method and a system that ensure robust accuracy while simultaneously reducing computational and memory complexity when using fixed-point quantized neural networks. This will be achieved by efficiently centering and scaling the dynamic range of activations after the convolution / accumulation stage and by performing hardware-friendly requantization.

[0009] In one embodiment, a computer-implemented method includes capturing a plurality of images with a camera associated with a vehicle and storing image data associated with the plurality of images in a memory. The method further includes assigning vehicle perception tasks to a plurality of processing elements connected to an accelerator associated with the memory. The method also includes at least one of the plurality of processing elements performing the vehicle perception tasks using a neural network, wherein the performance of the vehicle perception tasks includes quantizing a fixed-point value based on an activation input and a synapse weight.The procedure also includes controlling the vehicle, based at least partially on a result of performing the vehicle perception tasks.

[0010] The procedure also includes performing batch normalization to a fixed-point value. Batch normalization for non-zero values ​​is performed using the expression Σ w̃. ik x k + b̃ is performed and fed to an AND gate as a sign section and a mantissa section. An output of the AND gate is fed into a shift register to perform re-quantization for the dynamic range, and the results of the re-quantization are generated as output (512), where w̃ ik = γw ik and b̃ = b + β, where γ and β are parameters of a batch normalization (BN) layer, and where w ikLet y be a synaptic weight and b a systematic error. In some examples, batch normalization is performed to determine yi as follows: yi=ReLU[BN[∑wikxk+b]]={0if∑w˜ikxk+b˜<0∑w˜ikxk+b˜,otherwise where w̃ ik = γw ik and b̃ = b + β, where y and β are parameters from a batch normalization layer of the neural network. In some examples, performing batch normalization involves performing a 1-bit left shift of the fixed-point value using a shift register. In some examples, the fixed-point value is an 8-bit fixed-point value. In some examples, quantizing the fixed-point value involves converting a floating-point value to the fixed-point value. In some examples, the quantized fixed-point value is expressed as follows: Q(x)={−2IL−1,x≤−2IL−12IL−1−ε,x≥2IL−1−ε⌊x⌋,⌊x⌋≤x≤⌊x⌋+ε2⌊x⌋+ε,⌊x⌋+ε2 <x≤⌊x⌋+ε where IL is determined as floor(log2 3σ), where σ is the standard deviation of a probability density function, FL as 8 - IL, ε = 2 -FL is determined and ⌊x⌋ The largest multiplier of ε is a measure of accuracy. In some examples, the accelerator further includes an instruction taker for a neural processing unit, an allocation unit, a Level 1 cache, and a Level 2 cache. In some examples, the accelerator is communicatively coupled to a central processing unit of the main computer via memory, where the accelerator is a dedicated processor and the central processing unit of the main computer is a general-purpose processor. In some examples, the central processing unit of the main computer receives the image data from the camera associated with the vehicle and stores the image data associated with the multitude of images in memory.

[0011] In another embodiment, a system includes a memory containing computer-readable instructions and a processing device for executing the computer-readable instructions to perform a method. The method includes capturing a plurality of images with a camera attached to a vehicle and storing image data associated with the plurality of images in a memory. The method further includes assigning vehicle perception tasks to a plurality of processing elements connected to an accelerator associated with the memory.The method further includes performing vehicle perception tasks using a neural network through at least one of the plurality of processing elements, wherein performing the vehicle perception tasks includes quantizing a fixed-point value based on an activation input and a synaptic weight. The method further includes controlling the vehicle based at least partially on a result of performing the vehicle perception tasks.

[0012] In addition to one or more of the features described herein, the procedure in some examples also includes batch normalization to a fixed-point value. In some examples, batch normalization is performed to determine yi as follows: yi=ReLU[BN[∑wikxk+b]]={0if∑w˜ikxk+b˜<0∑w˜ikxk+b˜,otherwise where w̃ ik = γw ikand b̃ = b + β, where y and β are parameters from a batch normalization layer of the neural network. In some examples, performing batch normalization involves performing a 1-bit left shift of the fixed-point value using a shift register. In some examples, the fixed-point value is an 8-bit fixed-point value. In some examples, quantizing the fixed-point value involves converting a floating-point value to the fixed-point value. In some examples, the quantized fixed-point value is expressed as follows: Q(x)={−2IL−1,x≤−2IL−12IL−1−ε,x≥2IL−1−ε⌊x⌋,⌊x⌋≤x≤⌊x⌋+ε2⌊x⌋+ε,⌊x⌋+ε2 <x≤⌊x⌋+ε where IL is determined as floor(log2 3σ), where σ is the standard deviation of a probability density function, FL as 8 - IL, ε = 2 -FL is determined and ⌊x⌋ The largest multiplier of ε is a measure of accuracy. In some examples, the accelerator further includes an instruction taker for a neural processing unit, an allocation unit, a Level 1 cache, and a Level 2 cache. In some examples, the accelerator is communicatively coupled to a central processing unit of the main computer via memory, where the accelerator is a dedicated processor and the central processing unit of the main computer is a general-purpose processor. In some examples, the central processing unit of the main computer receives the image data from the camera associated with the vehicle and stores the image data associated with the multitude of images in memory.

[0013] In yet another embodiment, a computer program product includes a computer-readable storage medium containing program instructions, wherein the program instructions are executable by a processing device to cause the processing device to perform a method. A method includes capturing a plurality of images with a camera attached to a vehicle and storing image data associated with the plurality of images in a memory. The method further includes assigning vehicle perception tasks to a plurality of processing elements connected to an accelerator associated with the memory.The method further includes performing vehicle perception tasks using a neural network through at least one of the plurality of processing elements, wherein performing the vehicle perception tasks includes quantizing a fixed-point value based on an activation input and a synaptic weight. The method further includes controlling the vehicle based at least partially on a result of performing the vehicle perception tasks.

[0014] Other features, advantages and details appear only as examples in the following detailed description of the embodiments, the detailed description referring to the drawings in which: Fig. 1 represents a vehicle with a processing system for fixed-point quantization in neural networks according to one or more embodiments described herein; Fig. 2 the processing system of Fig. 1 represents fixed-point quantization in neural networks according to one or more embodiments described herein; Fig. 3A represents a 32-bit floating-point value that includes a 1-bit sign component, an 8-bit exponent component and a 23-bit mantissa component according to one or more embodiments described herein; Fig. 3B represents an 8-bit fixed-point value that includes a 1-bit sign component and a 7-bit mantissa component according to one or more embodiments described herein; Fig. 4. A block diagram for executing instructions on one or more of the processing elements of Fig. 2 represents in a single layer of a NN, wherein, according to one or more embodiments described herein, fixed-point values ​​are used instead of floating-point values; Fig. 5 represents a block diagram of a hardware implementation of a requantization technique according to one or more embodiments described herein; Fig. 6A represents an 8-bit fixed-point value with a 1-bit sign component and a 7-bit mantissa component according to one or more embodiments described herein; Fig. 6B represents an 8-bit fixed-point value resulting from a requantization technique according to one or more embodiments described herein; Fig. 7 represents a diagram of the weight distribution of the batch normalization according to one or more embodiments described herein; Fig. 8 represents a flowchart of a method for fixed-point quantization in neural networks according to one or more embodiments described herein; and Fig. Figure 9 represents a block diagram of a processing system for implementing the techniques described herein in accordance with the aspects of the present disclosure.

[0015] The technical solutions described herein provide vehicle perception using deep neural networks. Vehicle perception enables the detection and acquisition of objects by processing images captured by one or more cameras attached to a vehicle (e.g., a car, motorcycle, boat, or other vehicle type). Vehicle perception assists a driver / operator by providing information about the vehicle's exterior and / or assists autonomous vehicles by providing information useful for making driving decisions (e.g., whether to accelerate, brake, turn, etc.).

[0016] Modern vehicles generally include one or more cameras that provide additional support, capture images of the driver to determine their drowsiness or attentiveness, (while driving) provide visual recording of road traffic for collision avoidance purposes, and enable the recognition of structures such as road signs. For example, a vehicle may be equipped with multiple cameras, and images from several cameras (referred to as "surround-view cameras") can be used to create a "surround" or "bird's-eye view" of the vehicle. Some of the cameras (called "long-range cameras") can be used to capture images over long distances (e.g., for object detection for collision avoidance, structure detection, etc.).

[0017] These vehicles may also be equipped with an in-vehicle display (e.g., a touchscreen) used to show camera images and / or other images to the driver. For example, a conventional rearview and / or side mirror may be replaced by a display showing a camera image from a camera positioned at the rear of the vehicle, providing the driver with a "rear view" instead of the conventional rearview mirror view.

[0018] An example of such a vehicle is in Fig. 1 shown. In particular, it shows Fig. Figure 1 represents a vehicle 100 with a processing system 110 for fixed-point quantization in neural networks. The vehicle 100 can be a passenger car, a truck, a minibus, a bus, a motorcycle, or another suitable vehicle. The processing system 110 associated with the vehicle 100 is responsible for vehicle perception by analyzing images captured by a variety of cameras associated with the vehicle 100.

[0019] In the example of Fig. 1. The vehicle 100 includes cameras 120, 121, 122, 123 and cameras 130, 131, 132, 133. According to one or more embodiments described herein, cameras 120-123 are surround-view cameras that capture images outside and in the immediate vicinity of the vehicle 100. The images captured by cameras 120-123 together form a surround view (sometimes referred to as a "top-down view" or "bird's-eye view") of the vehicle 100. These images can be used for operating the vehicle (e.g., parking, reversing, etc.). According to one or more embodiments described herein, cameras 130-133 are wide-view cameras that, in contrast to cameras 120-123, capture images of the vehicle's exterior and of locations far away from the vehicle 100. These images can be processed using vehicle perception techniques and deep neural networks.The results can be used, for example, for object detection and avoidance. It should be noted that, although eight cameras are shown (120-123 and 130-133), more or fewer cameras can be implemented in different embodiments.

[0020] The captured images can be displayed on a screen (not shown) to provide the driver / operator of vehicle 100 with external views of the vehicle. The captured images can be displayed as live images, still images, or a combination of both. In some examples, the images can be combined into a composite view, such as a surround view. The displayed images can also be overlaid with information determined during processing / analysis. For example, an image can be overlaid with distance information (e.g., distance from vehicle 100 to another object / vehicle), safety information (e.g., a warning of a potential collision), and the like.

[0021] The vehicle 100 can also include sensors, such as sensors 140 and 141. According to one or more embodiments, sensor 140 is a high-resolution radar, and sensor 141 is a lidar (light imaging detection and range) sensor. Other numbers and types of sensors can also be used.

[0022] The processing system 110 uses images captured by one or more of the cameras 120-123, 130-133 to perform vehicle perception using deep neural networks. Vehicle perception can thus include, for example, performing feature extraction, object detection and avoidance, and the like.

[0023] The various components, modules, machines, etc., that are in Fig. The components described in Section 1 can be implemented as instructions stored on a computer-readable storage medium, as hardware modules, as purpose-built hardware (e.g., application-specific hardware, application-specific integrated circuits (ASICs), application-specific special-purpose processors (ASSPs), field-programmable gate arrays (FPGAs), as embedded controllers, hard-wired circuits, etc.), or as a combination or combinations thereof. According to the aspects of this disclosure, the machine (or machines) described herein can be a combination of hardware and programming. The programming can be processor-executable instructions stored on tangible memory, and the hardware can be a processing device (e.g., the CPU 921 of Fig. 9) to execute these instructions. Thus, a system memory (e.g., the RAM 924 of Fig. 9) Store program instructions which, when executed by the processing device, implement the machines described herein. Other machines may also be used to integrate other features and functions described in other examples herein.

[0024] Fig. 2 represents the processing system 110 of Fig. 1 for fixed-point quantization in neural networks according to one or more embodiments described herein. The processing system 110 receives images from one or more of the cameras 120-123, 130-133 at a processing device 202. The processing device 202 (also referred to as the central host processing unit) can perform object-level tracking, run applications, and perform other general-purpose tasks. Thus, the processing device 202 is a general-purpose hardware device. The processing device 202 is configured to receive the images from the cameras 120-123, 130-133 and store the images as image data in a memory 204 (also referred to as a shared memory), which may be a dynamic random-access memory or another suitable memory.

[0025] A dedicated hardware device, referred to as a "neural processing unit" or "accelerator" 210, is configured to receive the image data stored in memory 204 and to perform image processing on the image data. The image processing may include point cloud processing, neural network tasks, and the like. The accelerator 210 may be an ASIC, a field-programmable gate array (FPGA), a graphics processing unit (GPU), or another suitable dedicated hardware device.

[0026] According to one or more embodiments described herein, the accelerator 210 can utilize machine learning to achieve the various image processing / analysis operations described herein. More specifically, the accelerator 210 can integrate and utilize rule-based decision-making and artificial intelligence (AI) to achieve the various functionalities of the accelerator 210 described herein. The term "machine learning" broadly describes a function of electronic systems that learn from data.A machine learning system, machine, or module can include a trainable machine learning algorithm, which can be trained, for example, in an external cloud environment, to learn functional relationships between previously unknown inputs and outputs. The resulting model can then be used to perform feature extraction, object detection, collision avoidance, and the like. In one or more embodiments, the machine learning functionality can be implemented using an artificial neural network (ANN), which has the capability to be trained to perform a previously unknown function. In the fields of machine learning and cognitive science, ANNs are a family of statistical learning models inspired by the biological neural networks of animals, and particularly the brain.ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs.

[0027] ANNs can be implemented as so-called "neuromorphic" systems of interconnected processor elements that function as simulated "neurons" and exchange "messages" in the form of electronic signals. Similar to the "plasticity" of synaptic neurotransmitter connections that transmit messages between biological neurons, the connections in ANNs, which transmit electronic messages between simulated neurons, are assigned numerical weights corresponding to the strength or weakness of a particular connection. These weights can be adjusted and fine-tuned based on experience, allowing the ANNs to adapt to inputs and become capable of learning. For example, an ANN for recognizing and capturing objects is defined by a set of input neurons that can be activated by the pixels of an input image.After weighting and transformation by a function defined by the network designer, the activation of these input neurons is passed on to other downstream neurons, often referred to as "hidden" neurons or "hidden" layers. This process is repeated until an output neuron is activated. The activated output neuron determines which object was detected.

[0028] The accelerator 210 includes an instruction retriever 212 for the neural processing unit (NPU) for retrieving / receiving instructions and image data from memory 204. The instructions can include instructions for performing, for example, feature extraction, object recognition, and the like. The accelerator 210 also includes an assigner 214 for assigning / scheduling the instructions to a variety of processing elements 220.

[0029] The accelerator 210 also includes additional memory, such as cache memory 216, which can be a Level 2 (L2) cache, and a global memory buffer 218, which can be a Level 1 (L1) cache. The global memory buffer 218 can, for example, be a static random-access memory.

[0030] The processing elements 220 execute the vehicle perception techniques based on deep neural networks. In particular, the distributor 214 sends instructions and image data to the individual processing elements 220, which together process / analyze the image data to perform vehicle perception.

[0031] The processing elements typically use 220 floating-point operations to execute instructions allocated by the 214 assignor. Fig. For example, 3A represents a 32-bit floating-point value 300, which includes a 1-bit sign component 301, an 8-bit exponent component 302, and a 23-bit mantissa component 303 according to one or more embodiments described herein. The example of Fig. 3A represents the value -1.4212e-13.

[0032] A significant reduction in processing resources and the need for processing elements 220 can be achieved by using fixed-point values ​​instead of floating-point values. Fig. 3B represents an 8-bit fixed-point value 310, which includes a 1-bit sign component 311 and a 7-bit mantissa component 313, which is further subdivided into a 4-bit integer component 313a and a 3-bit fractional component 313b. The example of Fig. 3B represents the value 12.75. The use of fixed-point values ​​instead of floating-point values ​​can lead to an approximately twenty-fold (20x) reduction in the computing power applied by the processing elements 220, which represents a significant improvement in computer functionality.

[0033] The processing elements 220 can perform a quantization technique on a layer of the NN to obtain the fixed-point value, as in Fig. 4 is shown, to be output. In particular, it shows Fig. 4 a block diagram 400 for executing instructions on one or more of the processing elements 220 of Fig. 2 in a single layer of a NN, wherein, according to one or more embodiments described herein, fixed-point values ​​are used instead of floating-point values. For example, an instruction may be executed in a convolutional layer of the NN and take the following form: yi=∑wikxk+b, where yi The output is x k The activation input is, w ik the synapse weight and b the systematic error.

[0034] First, at block 406, block diagram 400 multiplies the value of the synapse weight wik (n bits) 402 and the value of the activation input xk (n bits) 404. The result 408 of the multiplication is in the form of 2n bits and is input into an accumulator machine 414. At block 410, the result 408 can be summed with a feedback signal 418, which is output 416 by the accumulator machine 414 and fed back into the accumulator machine 414 as input 412. The feedback signal 418 has the form 2N+M bits, where M is determined based on a largest kernel size (e.g., for a 5×5 filter, M=5). The output 416 is fed to a quantization machine 420, and a quantized output 422 is generated in the form of n bits.

[0035] The 420 quantization machine converts the floating-point value into a fixed-point value. For example, the 420 quantization machine converts, with respect to the Fig. 3A and Fig. 3B converts the floating-point representation 300, which is a 32-bit value, into a fixed-point representation as an 8-bit value. The quantization is performed based on the following expression to express a fixed-point value x as a quantum value Q(x): x→Q(x),st min∫−∞+∞|x−Q(x)|2dx

[0036] For uniform 8-bit quantization<IL, FL> , where IL is the bit length for the integer part of the value representing a circumference / range and FL is the bit length for the decimal (float) part of the value representing a position, the quantum value is expressed as follows: Q(x)={−2IL−1,x≤−2IL−12IL−1−ε,x≥2IL−1−ε⌊x⌋,⌊x⌋≤x≤⌊x⌋+ε2⌊x⌋+ε,⌊x⌋+ε2 <x≤⌊x⌋+ε where ε is a value for precision.

[0037] According to one or more embodiments described herein, IL is determined as floor(log2 3σ), where σ is the standard deviation of a probability density function, FL is determined as 8 - IL, ε = 2 -FL determined and is ⌊x⌋ the largest multiplier of ε.

[0038] The 220 processing elements can also perform batch normalization to a fixed-point value. Batch normalization centers the dynamic range with a fixed standard deviation. This allows for re-quantization to maintain the desired precision. Fig. Figure 5 shows a block diagram of a hardware implementation 500 of the requantization according to one or more embodiments described herein.

[0039] Batch normalization can be integrated into the convolutional layer of the neural network. In such cases, batch normalization is performed to determine yi using the following expression: yi=ReLU[BN[∑wikxk+b]]={0if∑w˜ikxk+b˜<0∑w˜ikxk+b˜,otherwise

[0040] In this example, w̃ ik = γw ik and b̃ = b + β, where y and β are the parameters of the batch normalization (BN) layer.

[0041] With reference to Fig. 5. Batch normalization for non-zero values ​​is performed at block 502 using the expression Σ w̃. ik x k+ b̃ is performed. Output 504 is fed to an AND gate 506 as a sign section 504a and a mantissa section 504b. The output 508 of the AND gate 506 is fed into a shift register 510 to perform re-quantization for the dynamic range, and the results of the re-quantization are generated as output 512.

[0042] The shift register 510, as in the Fig. 6A and Fig. Figure 6B shows a 1-bit left shift (L-shift). In particular, it represents Fig. 6A represents an 8-bit fixed-point value 600 with a 1-bit sign component 601 and a 7-bit mantissa component 602. The mantissa component 602 is further segmented into an integer component 602a and a fractional (decimal) component 602b. The integer component 602a is a 3-bit segment of the mantissa 602, and the fractional component 602b is a 4-bit segment of the mantissa 602. The shift register 510 acts on the fixed-point value 600 to perform the 1-bit left shift.

[0043] The output or results of the 1-bit left shift are displayed in Fig. 6B is represented, which is an 8-bit fixed-point value 610 with a 1-bit sign component 611 and a 7-bit mantissa component 612. The mantissa component 612 is further segmented into an integer component 612a and a fractional (decimal) component 612b. After the shift, the 3-bit integer component 602a is derived from Fig. 6A is converted into a 4-bit integer component 612a of the mantissa 612. Similarly, the 4-bit fractional component 602b is converted into a 3-bit fractional component 612b of the mantissa 612. Before batch normalization, the 8-bit fixed-point value 600 represents the value -8.0; 7.975. After batch normalization, the 8-bit fixed-point value 610 represents the value -4.0; 3.9375.

[0044] Fig. Figure 7 presents a diagram 700 of the weight distribution of the batch normalization according to one or more embodiments described herein. In this example, diagram 700 plots the weight distribution as a function of the parameter value (x-axis) against a probability density function (PDF) (y-axis). Minimum and maximum values ​​are also shown.

[0045] Fig. Figure 8 presents a flowchart of a method 800 for fixed-point quantization in neural networks according to one or more embodiments described herein. The method 800 can be performed by any suitable processing system and / or processing device, such as the processing system 110 from [reference missing]. Fig. 1 and Fig. 2, the accelerator 210 from Fig. 2, the processing system 900 from Fig. 9 or another suitable processing device and / or processing system.

[0046] In block 802, a camera assigned to vehicle 100 (e.g., one of cameras 120-123, 130-133) records a multitude of images. The processing device 202 stores image data associated with the multitude of images in the memory 204.

[0047] At block 804, the NPU instruction caller 212 retrieves vehicle perception tasks that indicate how the multitude of images should be processed, and the assigner 214 assigns vehicle perception tasks to the processing elements 220 of the accelerator 210 associated with the memory 204.

[0048] In block 806, at least one of the multiple processing elements 220 performs the vehicle perception tasks using a neural network. Specifically, the processing elements 220 perform the vehicle perception tasks by quantizing a fixed-point value based on an activation input and a synapse weight.

[0049] In Block 808, the results of the vehicle perception tasks are used to control vehicle 100. For example, an autonomous vehicle can be controlled to avoid an object detected as a result of the vehicle perception task. In another example, a driver of vehicle 100 can be alerted to an object detected as a result of the vehicle perception task.

[0050] Additional processes can also be included, and it is understood that the in Fig. The process shown in section 8 is for illustrative purposes only, and other processes may be added or existing processes removed, modified or rearranged without deviating from the scope of protection and spirit of the present revelation.

[0051] It is understood that the present revelation can be implemented in conjunction with any other type of computer environment now known or later developed. Fig.Figure 9 presents a block diagram of a processing system for implementing the techniques described herein. In examples, the processing system 900 includes one or more central processing units (processors) 921a, 921b, 921c, etc. (collectively or generally referred to as processor(s) 921 and / or processing device(s)). In aspects of this disclosure, each processor 921 may include a RISC (Reduced Instruction Set Computer) microprocessor. The processors 921 are coupled to the system memory (e.g., main memory (RAM) 924) and various other components via a system bus 933. The read-only memory (ROM) 922 is coupled to the system bus 933 and may include a basic input / output system (BIOS) that controls certain basic functions of the processing system 900.

[0052] Furthermore, an input / output (I / O) adapter 927 and a network adapter 926 coupled to the system bus 933 are shown. The I / O adapter 927 can be a SCSI adapter (“Small Computer System Interface”) that communicates with a hard disk 923 and / or a storage drive 925 or another similar component. The I / O adapter 927, hard disk 923, and storage device 925 are collectively referred to herein as mass storage 934. The operating system 940 for execution on the processing system 900 can be stored in the mass storage 934. The network adapter 926 connects the system bus 933 to an external network 936, enabling the processing system 900 to communicate with other such systems.

[0053] A display (e.g., a display monitor) 935 is connected to the system bus 933 via the display adapter 932, which may include a graphics adapter or video controller to improve the performance of graphics-intensive applications. In one aspect of this disclosure, the adapters 926, 927, and / or 932 may be connected to one or more I / O buses that are connected to the system bus 933 via an intermediate bus bridge (not shown). Suitable I / O buses for connecting peripheral devices such as aps disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI). Additional input / output devices are shown to be connected to the system bus 933 via the user interface adapter 928 and the display adapter 932.A keyboard 929, a mouse 930 and a speaker 931 can be connected to the system bus 933 via the user interface adapter 928, which can include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit.

[0054] In some aspects of the present disclosure, the processing system 900 includes a graphics processing unit 937. The graphics processing unit 937 is a special electronic circuit designed to manipulate and modify memory in order to accelerate the generation of images in a frame buffer intended for output on a display. In general, the graphics processing unit 937 is very efficient at manipulating computer graphics and image processing and has a highly parallel structure, which makes it more effective than general-purpose CPUs for algorithms where the processing of large blocks of data is performed in parallel.

[0055] Thus, the processing system 900, as configured herein, includes processing capacity in the form of processors 921, storage capacity including system memory (e.g., RAM 924) and mass storage 934, input means such as a keyboard 929 and mouse 930, and output capability including speakers 931 and a display 935. In some aspects of this disclosure, a portion of the system memory (e.g., RAM 924) and the mass storage 934 jointly store an operating system to coordinate the functions of the various components represented in the processing system 900.

Claims

[1] Computer-implemented method (800) for vehicle perception, the method comprising: Capturing (802) a multitude of images on a camera assigned to a vehicle (120, 121, 122, 123, 130, 131, 132, 133) and storing image data associated with the multitude of images in a memory (204); Assigning (804) vehicle perception tasks to a plurality of processing elements (220) of an accelerator (210) connected to the memory (204); Performing (806) the vehicle perception tasks for vehicle perception using a neural network by at least one of the plurality of processing elements (220), wherein performing the vehicle perception tasks includes quantizing a fixed-point value based on an activation input (x_k) and a synapse weight (w_ik); Controls (808) of the vehicle (100), based at least in part on a result of performing the vehicle perception tasks; and Performing batch normalization to the fixed-point value, where batch normalization for non-zero values ​​is performed using the expression Σ w̃ ik x k + b̃ is performed (502) and fed to an AND gate (506) as a sign section (504a) and a mantissa section (504b), wherein an output (508) of the AND gate (506) is fed into a shift register (510) to perform a re-quantization for the dynamic range, and the results of the re-quantization are generated as output (512), where w̃ ik = γw ik and b̃ = b + β, where y and β are parameters of a batch normalization (BN) layer, and where w ik a synapse weight and b is a systematic error. [2] Computer-implemented method (800) according to claim 1, wherein batch normalization is performed to determine y i to be determined as follows: yi=ReLU[BN[∑wikxk+b]]={0if∑w˜ikxk+b˜<0∑w˜ikxk+b˜,otherwise where w̃ ik = γw ik and b̃ = b + β, where y and β are parameters from a batch normalization layer of the neural network. [3] Computer-implemented method (800) according to claim 1, wherein performing the batch normalization comprises performing a 1-bit left shift of the fixed-point value using a shift register. [4] Computer-implemented method (800) according to claim 1, wherein the fixed-point value is an 8-bit fixed-point value. [5] Computer-implemented method (800) according to claim 1, wherein quantizing the fixed-point value comprises converting a floating-point value to the fixed-point value. [6] Computer-implemented method (800) according to claim 1, wherein the quantized fixed-point value is expressed as follows: Q(x)={−2IL−1,x≤−2IL−12IL−1−ε,x≥2IL−1−ε⌊x⌋,⌊x⌋≤x≤⌊x⌋+ε2⌊x⌋+ε,⌊x⌋+ε2 <x≤⌊x⌋+ε where IL is determined as floor(log2 3σ), where σ is the standard deviation of a probability density function, FL as 8 - IL, ε = 2 -FL is determined and ⌊x⌋ a largest multiplier of ε is, which is a value of the accuracy. [7] Computer-implemented method (800) according to claim 1, wherein the accelerator (210) further comprises an instruction retriever (212) for a neural processing unit, an assigner (214), a level 1 cache memory (218) and a level 2 cache memory (216). [8] Computer-implemented method (800) according to claim 1, wherein the accelerator (210) is coupled via the memory (204) to a central processing unit (202) of the main computer, wherein the accelerator (210) is a dedicated processor, wherein the central processing unit (202) of the main computer is a general-purpose processor, and wherein the central processing unit (202) of the main computer receives the image data from the camera (120, 121, 122, 123, 130, 131, 132, 133) assigned to the vehicle (100) and stores the image data assigned to the plurality of images in the memory (204). [9] System (110) comprising: a memory (204) containing computer-readable instructions; and a processing device (202) for executing the computer-readable instructions for performing a vehicle sensing method, the method comprising the following: Capturing a large number of images on a camera (120, 121, 122, 123, 130, 131, 132, 133) assigned to a vehicle (100) and storing image data associated with the large number of images in a memory (204); Assigning vehicle perception tasks to a multitude of processing elements (220) of an accelerator (210) connected to the memory (204); Performing the vehicle perception tasks for vehicle perception using a neural network by at least one of the plurality of processing elements (220), wherein performing the vehicle perception tasks is the quantization of a fixed-point value based on an activation input (x k ) and a synaptic weight (w ik ) includes; Driving the vehicle (100), based at least in part on a result of performing the vehicle perception tasks; and Performing batch normalization to the fixed-point value, where batch normalization for non-zero values ​​is performed using the expression Σ W iu x u + b̃ is performed (502) and fed to an AND gate (506) as a sign section (504a) and a mantissa section (504b), wherein an output (508) of the AND gate (506) is fed into a shift register (510) to perform a re-quantization for the dynamic range, and the results of the re-quantization are generated as output (512), where w̃ ik = γw ik and b̃ = b + β, where y and β are parameters of a batch normalization (BN) layer, and where w ik a synapse weight and b is a systematic error.

Images