Methods for improving the accuracy of quantized multi-level object detection networks

Abstract

A device includes a memory and a processor. The memory can be configured to store image data of an input image. The processor can be configured to detect one or more objects in the input image using a quantized multi-level object detection network, wherein quantization of the quantized multi-level object detection network includes (i) generating quantized image data by performing a first data range analysis on the image data of the input image, (ii) generating a feature map and a proposal bounding box by applying a region proposal network (RPN) to the quantized image data, (iii) performing a region of interest pooling operation on the feature map and a plurality of ground truth boxes corresponding to the proposal bounding boxes generated by the RPN, (iv) generating a quantized region of interest pooling result by performing a second data range analysis on the result from the region of interest pooling operation, and (v) applying a region-based convolutional neural network (RCNN) to the quantized region of interest pooling result.

Description

Technical Field

[0001] The present invention relates generally to computer vision, and more particularly to methods and / or apparatus for improving the accuracy of quantized multi-stage object detection networks. Background Art

[0002] In order to minimize the accuracy degradation of the quantized neural network model after quantization, the quantization method analyzes the data range of the input data. The accuracy of the quantized model is very sensitive to the data (especially the input data) analyzed in the data range analysis.

[0003] Existing object detection networks rely on region proposal algorithms to hypothesize object locations. An existing region-based object detection network is Faster-RCNN. Faster-RCNN consists of a region proposal network (RPN) and a region-based convolutional neural network (RCNN). Faster-RCNN functions such as bbox_transform_inv, non-maximum suppression (NMS), and ROI pooling are performed between the RPN and RCNN. The RPN uses default anchors to obtain bounding box proposals that may contain the object while performing feature extraction on the image. The proposed bounding box then undergoes bbox transformation, NMS, and ROI pooling before entering the RCNN. Because these intermediate steps are not very friendly to artificial intelligence (AI) chips, industry practice is to split the network into two parts: the RPN and the RCNN. Since the network is split into two parts, data range analysis (DRA) needs to be performed twice. The first DRA is performed on the image presented as input to the RPN. For multi-stage object detection, the industry practice is to perform a second DRA on the output of ROI pooling. The data presented by ROI pooling is determined by the proposal bounding boxes generated by the RPN, which are presented as input to the RCNN. Although the data from ROI pooling is the same as inference, the data is less suitable for quantization.

[0004] It is desirable to implement a method to improve the accuracy of a quantized multi-stage object detection network. Summary of the Invention

[0005] The present invention covers aspects relating to an apparatus comprising a memory and a processor. The memory may be configured to store image data of an input image. The processor may be configured to detect one or more objects in the input image using a quantized multi-stage object detection network, wherein quantizing the quantized multi-stage object detection network comprises: (i) generating quantized image data by performing a first data range analysis on the image data of the input image; (ii) generating a feature map and a proposal bounding box by applying a region proposal network (RPN) to the quantized image data; (iii) performing a region of interest pooling operation on the feature map and a plurality of ground truth boxes corresponding to the proposal bounding boxes generated by the RPN; (iv) generating a quantized region of interest pooling result by performing a second data range analysis on the result from the region of interest pooling operation; and (v) applying a region-based convolutional neural network (RCNN) to the quantized region of interest pooling result.

[0006] In some embodiments of the apparatus aspects described above, the ground truth box is a projection of the proposal bounding box onto the input image.

[0007] In some embodiments of the apparatus aspects described above, the second data range analysis applies the technique applied by the first data range analysis.

[0008] In some embodiments of the apparatus aspects described above, training the RPN and RCNN includes: (i) generating feature maps and proposed bounding boxes by applying the RPN to image data of an input image; and (ii) performing a region of interest pooling operation on the feature maps and proposed bounding boxes generated by the RPN.

[0009] In some embodiments of the apparatus aspects described above, the RPN and RCNN are stored in the processor as directed acyclic graphs and corresponding weights.

[0010] In some embodiments of the apparatus aspects described above, the RPN and the RCNN share one or more convolutional layers.

[0011] In some embodiments of the apparatus aspects described above, the processor is further configured to generate a pooling result for the region of interest by cropping and resampling a corresponding portion of the feature map to which an object detection proposal is assigned. In some embodiments, the resampling includes a warp operation. In some embodiments, the processor can configure bilinear interpolation hardware to resize features of the region of interest for use as input to a subsequent subnetwork in hardware.

[0012] In some embodiments of the apparatus aspects described above, the feature map comprises a three-dimensional array having dimensions corresponding to a depth, a height, and a width of the feature map.

[0013] In some embodiments of the apparatus aspects described above, the memory and the processor are part of at least one of a computer vision system or an autonomous vehicle.

[0014] The present invention also covers aspects relating to a method for object detection, the method comprising: storing image data of an input image in a memory; and detecting one or more objects in the input image using a quantized multi-stage object detection network, wherein quantization of the quantized multi-stage object detection network is performed by: (i) generating quantized image data by performing a first data range analysis on the image data of the input image; (ii) generating a feature map and a proposal bounding box by applying a region proposal network (RPN) to the quantized image data; (iii) performing a region of interest pooling operation on the feature map and a plurality of ground truth boxes corresponding to the proposal bounding boxes generated by the RPN; (iv) generating a quantized region of interest pooling result by performing a second data range analysis on the result from the region of interest pooling operation; and (v) applying a region-based convolutional neural network (RCNN) to the quantized region of interest pooling result.

[0015] In some embodiments of the method aspects described above, the ground truth box is a projection of the proposal bounding box onto the input image.

[0016] In some embodiments of the method aspects described above, the method further includes training an RPN and an RCNN, wherein the training of the RPN and the RCNN includes: (i) generating feature maps and proposed bounding boxes by applying the RPN to image data of an input image; and (ii) performing a region of interest pooling operation on the feature maps and proposed bounding boxes generated by the RPN.

[0017] In some embodiments of the method aspects described above, the RPN and RCNN are stored in the processor as directed acyclic graphs and corresponding weights.

[0018] In some embodiments of the method aspects described above, the RPN and the RCNN share one or more convolutional layers.

[0019] In some embodiments of the method aspects described above, the method further comprises generating a pooling result for the region of interest by using a processor to perform cropping and resampling on a corresponding portion of the feature map to which the object detection proposal is assigned. In some embodiments, the resampling comprises applying a warping operation.

[0020] In some embodiments of the method aspects described above, the method further comprises configuring, using the processor, bilinear interpolation hardware to resize features of the region of interest for use as input to a subsequent subnetwork in hardware.

[0021] In some embodiments of the method aspects described above, the feature map comprises a three-dimensional array having dimensions corresponding to a depth, a height, and a width of the feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Embodiments of the present invention will become apparent from the following detailed description, appended claims, and accompanying drawings, in which:

[0023] Figure 1 is a diagram illustrating a context in which a quantized multi-stage object detection network according to an example embodiment of the present invention may be implemented.

[0024] Figure 2 It shows Figure 1 Figure 2 shows a general implementation of a hardware engine.

[0025] Figure 3 is a diagram illustrating a multi-stage object detection process according to an example embodiment of the present invention.

[0026] Figure 4 is a diagram illustrating an example region of interest (ROI) pooling operation implemented in the context of object detection using a convolutional neural network (CNN) according to an example embodiment of the present invention.

[0027] Figure 5 is a diagram illustrating a process during a training or inference phase according to an example embodiment of the present invention.

[0028] Figure 6 is a diagram illustrating a process during a quantization stage according to an example embodiment of the present invention.

[0029] Figure 7 It shows that it can be achieved Figure 3 A diagram of an example of a process on a computer system.

[0030] Figure 8 is a diagram illustrating a camera system illustrating an example implementation of a computer vision system in which a quantized multi-stage object detection network according to an example embodiment of the present invention may be utilized. DETAILED DESCRIPTION

[0031] Embodiments of the present invention include a method for improving the accuracy of a quantized multi-stage object detection network, which can: (i) directly use ground truth to perform a second-stage data range analysis (DRA) during quantization; (ii) reduce or eliminate the impact of excessive non-object proposal bounding boxes on the results of the second-stage data range analysis; (iii) provide a second-stage data range analysis independent of the first-stage; (iv) obtain an optimal distribution of data range analysis data; (v) obtain relatively high accuracy; (vi) be used in all networks including a region proposal network (RPN); (vii) be used in a cascade of networks (e.g., Faster-RCNN, Mask-RCNN, Cascade-RCNN, MTCNN, etc.); (viii) construct a convolutional feature map; (ix) assign proposals / ROIs to ground truth bounding boxes in the input image; (x) be implemented on embedded devices with limited hardware resources and power budget; (xi) be implemented using a vector processor, CPU, DSP or GPU and / or (xii) be implemented as one or more integrated circuits.

[0032] refer to Figure 1 , shows a diagram of processing circuitry 80 illustrating the context in which a detection process incorporating a region of interest (ROI) pooling method according to an example embodiment of the present invention may be implemented. In various embodiments, processing circuitry 80 may be implemented as part of a computer vision system. In various embodiments, processing circuitry 80 may be implemented as part of a camera, a computer, a server (e.g., a cloud server), a smartphone (e.g., a cellular phone), a personal digital assistant, and the like. In one example, processing circuitry 80 may be configured for applications including, but not limited to, autonomous and semi-autonomous vehicles (e.g., cars, trucks, motorcycles, agricultural machinery, drones, aircraft, etc.), manufacturing, and / or security and surveillance systems. In contrast to general-purpose computers, processing circuitry 80 typically includes hardware circuitry optimized to provide high-performance image processing and computer vision pipelines in a minimal area and with minimal power consumption. In examples, various operations for performing image processing, feature detection / extraction, and / or object detection / classification for computer (or machine) vision may be implemented using hardware modules designed to reduce computational complexity and efficiently use resources.

[0033] In an example embodiment, processing circuitry 80 may include block (or circuit) 82, block (or circuit) 84, block (or circuit) 86, and / or memory bus 88. Circuitry 82 may implement a first processor. Circuitry 84 may implement a second processor. In an example, circuitry 84 may implement a computer vision processor. In an example, processor 84 may be an intelligent vision processor. Circuitry 86 may implement external memory (e.g., memory external to circuitry 82 and 84). In an example, circuitry 86 may be implemented as a dynamic random access memory (DRAM) circuit. Processing circuitry 80 may include other components (not shown). The number, type, and / or arrangement of the components of processing circuitry 80 may vary depending on the design criteria of a particular implementation.

[0034] Circuit 82 may implement a processor circuit. In some embodiments, processor circuit 82 may be implemented using a general-purpose processor circuit. Processor 82 may be operable to interact with circuit 84 and circuit 86 to perform various processing tasks. In an example, processor 82 may be configured as a controller for circuit 84. Processor 82 may be configured to execute computer-readable instructions. In one example, the computer-readable instructions may be stored by circuit 86. In some embodiments, the computer-readable instructions may include controller operations. Processor 82 may be configured to communicate with circuit 84 and / or access results generated by components of circuit 84. In an example, processor 82 may be configured to utilize circuit 84 to perform operations associated with one or more neural network models.

[0035] In an example, the processor 82 can be configured to program the circuit 84 with one or more pre-trained artificial neural network models (ANNs), including a region proposal network (RPN) 100, a region-based convolutional neural network (RCNN) 102, and corresponding weights / kernels (WGTS) 104 associated with the RPN 100 and the RCNN 102. In various embodiments, the RPN 100 and the RCNN 102 can be configured (trained and / or quantized) for operation in an edge device. In an example, the processing circuit 80 can be coupled to a sensor (e.g., a video camera, etc.) configured to generate data input. The processing circuit 80 can be configured to generate one or more outputs in response to the data input from the sensor based on one or more inferences made by executing the RPN 100 and the RCNN 102 using the corresponding weights / kernels (WGTS) 104. The operations performed by the processor 82 can vary according to the design criteria of a particular implementation.

[0036] In various embodiments, circuitry 86 may implement dynamic random access memory (DRAM) circuitry. Circuitry 86 is generally operable to store multi-dimensional arrays of input data elements and various forms of output data elements. Circuitry 86 may exchange input data elements and output data elements with processor 82 and processor 84.

[0037] The processor 84 can implement a computer vision processor circuit. In an example, the processor 84 can be configured to implement various functions for computer vision. The processor 84 is generally operable to perform specific processing tasks arranged by the processor 82. In various embodiments, all or part of the processor 84 can be implemented only in hardware. The processor 84 can directly execute the data stream generated by the software (e.g., directed acyclic graph, etc.) for object detection with region of interest pooling and specified processing (e.g., computer vision) tasks. In some embodiments, the processor 84 can be a representative example of several computer vision processors implemented by the processing circuit 80 and configured to operate together.

[0038] In an example embodiment, processor 84 generally includes block (or circuit) 90, one or more blocks (or circuits) 92a-92n, block (or circuit) 90, path 96, and block (or circuit) 98. Block 90 may implement a scheduler circuit. Blocks 92a-92n may implement hardware resources (or engines). Block 94 may implement a shared memory circuit. Block 98 may implement a directed acyclic graph (DAG) memory. In an example embodiment, one or more of circuits 92a-92n may include blocks (or circuits) 110a-110n. In the example shown, circuits 110a and 110b are implemented.

[0039] In an example embodiment, circuit 110a can implement a convolution operation. In another example, circuit 110b can be configured to provide a pooling operation. Circuit 110b can be configured to provide a ROI pooling operation according to an example embodiment of the present invention (e.g., using ground truth input as part of a quantization process). Convolution and pooling operations can be used to perform computer (or machine) vision tasks (e.g., as part of an object detection process, etc.). In yet another example, one or more of circuits 92c-92n may include blocks (or circuits) 110c-110n (not shown) to provide multi-dimensional convolution calculations.

[0040] In an example, circuitry 84 may be configured to receive a directed acyclic graph (DAG) from processor 82. The DAG received from processor 82 may be stored in DAG memory 98. Circuitry 84 may be configured to execute the DAG for RPN 100 and RCNN 102 using circuits 90, 92a-92n, and 94.

[0041] A plurality of signals (e.g., OP_A to OP_N) may be exchanged between circuit 90 and corresponding circuits 92a-92n. Each signal OP_A to OP_N may convey information about an execution operation and / or a generation operation. A plurality of signals (e.g., MEM_A to MEM_N) may be exchanged between corresponding circuits 92a-92n and circuit 94. Signals MEM_A to MEM_N may carry data. Signals (e.g., DRAM) may be exchanged between circuit 86 and circuit 94. Signal DRAM may transmit data between circuits 86 and 90 (e.g., on memory bus 96).

[0042] Circuit 90 may implement a scheduler circuit. Scheduler circuit 90 is generally operable to schedule tasks between circuits 92a-92n to perform various computer vision-related tasks defined by processor 82. Scheduler circuit 90 may assign individual tasks to circuits 92a-92n. Scheduler circuit 90 may assign individual tasks in response to parsing a directed acyclic graph (DAG) provided by processor 82. Scheduler circuit 90 may time-multiplex tasks to circuits 92a-92n based on the availability of circuits 92a-92n to perform the work.

[0043] Each circuit 92a-92n can implement a processing resource (or hardware engine). The hardware engines 92a-92n are generally operable to perform specific processing tasks. The hardware engines 92a-92n can be implemented to include dedicated hardware circuits that are optimized to have high performance and low power consumption when performing specific processing tasks. In some configurations, the hardware engines 92a-92n can operate in parallel and independently of each other. In other configurations, the hardware engines 92a-92n can operate together with each other to perform assigned tasks.

[0044] The hardware engines 92a-92n may be homogeneous processing resources (e.g., all circuits 92a-92n may have the same capabilities) or heterogeneous processing resources (e.g., two or more circuits 92a-92n may have different capabilities). The hardware engines 92a-92n are generally configured to execute operators, which may include, but are not limited to, resampling operators, warp operators, component operators that manipulate lists of components (e.g., components may be regions of a vector that share common properties and may be grouped together with bounding boxes), matrix inverse operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, min-max-reduce operators, pooling operators, non-min-non-max suppression operators, aggregation operators, scattering operators, statistical operators, classifier operators, integral image operators, upsampling operators, and downsampling operators that are powers of two, among others.

[0045] In various embodiments, the hardware engines 92a-92n may be implemented as hardware circuits only. In some embodiments, the hardware engines 92a-92n may be implemented as general-purpose engines that can be configured to operate as special-purpose machines (or engines) through circuit customization and / or software / firmware. In some embodiments, the hardware engines 92a-92n may be alternatively implemented as one or more instances or threads of program code executed on the processor 82 and / or one or more processors 84 (including but not limited to vector processors, central processing units (CPUs), digital signal processors (DSPs), or graphics processing units (GPUs)). In some embodiments, the scheduler 90 may select one or more of the hardware engines 92a-92n for a particular process and / or thread. The scheduler 90 may be configured to assign the hardware engines 92a-92n to a particular task in response to parsing the directed acyclic graph stored in the DAG memory 98.

[0046] Circuit 94 may implement a shared memory circuit. Shared memory 94 may be configured to store data in response to input requests and / or present data in response to output requests (e.g., requests from processor 82, DRAM 86, scheduler circuit 90, and / or hardware engines 92a-92n). In an example, shared memory circuit 94 may implement on-chip memory for computer vision processor 84. Shared memory 94 is generally operable to store all or part of a multidimensional array (or vector) of input data elements and output data elements generated and / or utilized by hardware engines 92a-92n. Input data elements may be transferred from DRAM circuit 86 to shared memory 94 via memory bus 88. Output data elements may be sent from shared memory 94 to DRAM circuit 86 via memory bus 88.

[0047] Path 96 may implement a transfer path internal to processor 84. Transfer path 96 is generally operable to move data from scheduler circuitry 90 to shared memory 94. Transfer path 96 may also be operable to move data from shared memory 94 to scheduler circuitry 90.

[0048] The processor 82 is shown in communication with the computer vision processor 84. The processor 82 can be configured as a controller for the computer vision processor 84. In some embodiments, the processor 82 can be configured to transmit instructions to the scheduler 90. For example, the processor 82 can provide one or more directed acyclic graphs to the scheduler 90 via the DAG memory 98. The scheduler 90 can initialize and / or configure the hardware engines 92a-92n in response to parsing the directed acyclic graphs. In some embodiments, the processor 82 can receive status information from the scheduler 90. For example, the scheduler 90 can provide status information and / or the readiness of outputs from the hardware engines 92a-92n to the processor 82 to enable the processor 82 to determine the next one or more instructions to execute and / or decisions to make. In some embodiments, the processor 82 can be configured to communicate with the shared memory 94 (e.g., directly or through the scheduler 90, which receives data from the shared memory 94 via the path 96). The processor 82 can be configured to retrieve information from the shared memory 94 to make decisions. The instructions executed by processor 82 in response to information from computer vision processor 84 may vary according to the design criteria of a particular implementation.

[0049] Circuit 110a can implement a convolution circuit. Convolution circuit 110a can communicate with memory 94 to receive input data and present output data. Convolution circuit 110a is generally operable to fetch multiple data vectors from shared memory circuit 94. Each data vector can include multiple data values. Convolution circuit 110a can also be operable to fetch a kernel from shared memory 94. The kernel typically includes multiple kernel values. Convolution circuit 110a can also be operable to fetch a block from shared memory 94 to an internal (or local) buffer. The block typically includes multiple input tiles. Each input tile can include multiple input values ​​across multiple dimensions. Convolution circuit 110a can also be operable to calculate multiple intermediate values ​​in parallel by multiplying each input tile in the internal buffer by a corresponding one of the kernel values, and to calculate an output tile including multiple output values ​​based on the intermediate values. In various embodiments, convolution circuit 110a can be implemented solely in hardware. An example of a convolution computation scheme that may be used to implement circuit 110a may be found in US Pat. No. 10,210,768, which is incorporated herein by reference in its entirety.

[0050] The circuit 110b can implement a pooling process according to an embodiment of the present invention. In various embodiments, the region of interest pooling scheme according to an embodiment of the present invention can be performed according to the implementation description provided herein.

[0051] refer to Figure 2 , shows the description Figure 1FIG2 is a diagram of an example implementation of a general hardware engine 92x. Hardware engine 92x may represent hardware engines 92a-92n. Hardware engine 92x generally includes a block (or circuit) 120, a block (or circuit) 122, a block (or circuit) 124, and a plurality of blocks (or circuits) 126a-126n. Circuit 120 may be implemented as a memory (or buffer) pair 120a and 120b. Circuit 122 may implement a controller circuit. In an example, circuit 122 may include one or more finite state machines (FSMs) configured to control various operators implemented by hardware engine 92x. Circuit 124 may implement a processing pipeline for hardware engine 92x. Circuits 126a-126n may implement a first-in, first-out (FIFO) memory. Circuits 126a-126n may be configured as input buffers for processing pipeline 124. Shared memory 94 may be configured (eg, by signals from circuitry 122 ) as a plurality of shared input buffers 128 a - 128 n and one or more output buffers 130 .

[0052] Signals (e.g., ADDR / CONFIG) may be generated by scheduler circuitry 90 and received by hardware engine 92x. Signal ADDR / CONFIG may carry address information and configuration data. Signals (e.g., BUSY_LEVEL) may be generated by circuitry 122 and transmitted to scheduler circuitry 90. Signal BUSY_LEVEL may convey the busy level of hardware engine 92x. Signals (e.g., STATUS / TARGETS) may be generated by circuitry 122 and transmitted to scheduler circuitry 90. Signal STATUS / TARGETS may provide status information about hardware engine 92x and target information for operands.

[0053] In an example embodiment, buffers 120a and 120b can be configured as dual-bank configuration buffers. The dual-bank buffers can be operable to store configuration information for a currently running operation in one buffer (e.g., buffer 120b) while simultaneously moving configuration information for the next operation to the other buffer (e.g., buffer 120a). Scheduler 90 typically loads operator configuration information (including status words in cases where an operator has already been partially processed in a previous operator block) into the dual-bank buffers. Once circuit 122 completes configuration information for the currently running operation and has received configuration information for the next operation, buffers 120a and 120b can be swapped.

[0054] Circuitry 122 typically implements control circuitry for hardware engine 92x. Circuitry 122 determines when to switch from a currently running operator to a new operator. Controller 122 is typically operable to control the movement of information into, out of, and within hardware engine 122x. Typically, the operation of hardware engine 92x is pipelined. During an operator switch, the front end of pipeline 124 may already be processing data for the new operator while the tail end of pipeline 124 is still completing processing associated with the old operator.

[0055] The circuitry 124 may implement pipeline circuitry. The pipeline circuitry 124 is generally operable to process operands received from the shared memory 94 using functions designed into the hardware engines 92x. The circuitry 124 may transfer data generated by the executed functions to one or more shared buffers 130.

[0056] Buffers 126a-126n may implement FIFO buffers. FIFO buffers 126a-126n may be operable to store operands received from shared buffers 128a-128n for processing in pipeline 124. In general, the number of FIFO buffers and the number of shared buffers implemented may vary to meet the design criteria of a particular application.

[0057] refer to Figure 3 , a diagram illustrating a multi-class object detection process according to an example embodiment of the present invention is shown. In an example embodiment, process (or method) 200 generally implements a multi-class object detection process according to an example embodiment of the present invention. In an example embodiment, process 200 may include steps (or states) 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, and 222. In step 202, process 200 may perform a feature extraction process on image data of one or more input images (IMAGE) using a region proposal network. During the training and inference phases of process 200, the feature extraction process may be applied directly to the image data of the input image(s). During the quantization phase of process 200, the feature extraction process may be performed on quantized image data of the input image(s). In various embodiments, the quantized image data of the input image(s) may be obtained by applying a first data range analysis (DRA) to the image data of the input image(s).

[0058] In step 204, the process 200 may generate a feature map based on the results from step 202. In step 206, the process 200 may predict a number of bounding boxes based on the results from step 202. In step 208, the process 200 may predict a corresponding class probability for each of the plurality of bounding boxes based on the results from step 202. In step 210, a plurality of proposal bounding boxes 214 may be generated by applying a bounding box transformation function (e.g., bbox_transform_inv) using as input the number of bounding boxes generated in step 206, the class probabilities generated in step 208, and a plurality of default anchors 212.

[0059] In step 216, a region of interest (ROI) pooling operation may be performed. The ROI pooling operation of step 216 is typically performed using the feature map generated in step 204 as one input and a second input selected in step 218. In step 218, the second input to the ROI pooling step 216 is typically selected based on whether process 200 is being performed in a training phase, a quantization phase, or an inference (operation) phase. In the training and inference phases, step 218 typically selects the proposal bbox 214 as the second input to the ROI pooling step 216 and directly presents the output of the ROI pooling step 216 as the input to step 220. In the quantization phase, step 218 typically selects the ground truth bbox 222 as the second input to the ROI pooling step 216 and performs a second data range analysis (DRA) on the output of the ROI pooling step 216 before presenting it as the input to step 220.

[0060] In step 220, process 200 may apply a region-based convolutional neural network (RCNN) model to the output of ROI pooling step 216 to identify whether one or more predetermined object categories are present in the input image, and if so, how many instances of the one or more predetermined object categories are present. In some embodiments, step 220 may provide (generate) two outputs, including the location of the bounding box and the probability of each of the one or more categories.

[0061] Because the proposal bbox 214 comes from the RPN, the proposal bbox 214 is typically offset relative to the ground truth bbox 222. However, this offset is typically not a problem because the RCNN model applied during step 220 typically also fine-tunes the position. In an example, the ground truth box 222 can be taken from annotations in some datasets such as VOC, COCO, etc. Annotations are typically labeled by individuals or organizations. In the example pipeline, the ground truth from the corresponding annotations can be read.

[0062] During the quantization phase, the ROI pooling operation 216 applies a second data range analysis (DRA) to generate a plurality of quantized proposal bounding boxes using the plurality of ground truth boxes 222 selected in step 218. Using the ground truth bboxes 222 generally improves the accuracy of the quantized data distribution from the ROI pooling step 216. The first data range analysis and the second data range analysis are generally implemented using the same algorithm. The difference is that the first data range analysis processes the input image data and the second data range analysis processes the output of the ROI pooling step 216. Generally, conventional or proprietary data range analysis algorithms can be used for the first data range analysis and the second data range analysis in the quantization phase.

[0063] In various embodiments, process 200 can be used during each of the steps or levels or stages of the normal artificial neural network development flow (training->quantization->deployment (real-time inference)). During training, the ROI pooling of process 200 typically uses a traditional approach using feature maps and proposal bboxes generated by a region proposal network (RPN). When training is complete, a quantization process can be performed. Quantization is typically performed based on the input data range to determine the optimal coefficients so that fixed-point values ​​can be used to more accurately replace floating-point values. During quantization, the ROI pooling of process 200 switches to a new approach that uses feature maps and ground truth bounding boxes generated by a region proposal network (RPN) and applies data range analysis to the ROI pooling result before presenting it to a region-based convolutional neural network (RCNN) model.

[0064] In the quantization stage, as many test scene pictures as possible are usually provided for data range analysis. The pictures of the training set can also be used here for quantization for fixed-point applications. During quantization according to an embodiment of the present invention, the ROI pooling of process 200 is performed by adopting the feature map and ground truth bounding box generated by the region proposal network (RPN) instead of the proposal bbox generated by the region proposal network (RPN). In the example, the ground truth box can be taken from the annotations in some data sets such as VOC, COCO, etc. The annotations are usually marked by individuals or organizations. In the example pipeline, the ground truth from the corresponding annotations can be read.

[0065] When quantization is complete, the network is ready to be deployed (e.g., inference operation). During inference operation, the ROI pooling of process 200 typically returns to the traditional method, thereby using the feature map and proposal bbox generated by the region proposal network (RPN). In various embodiments, RPN and RCNN can share one or more convolutional layers.

[0066] refer to Figure 4, a diagram illustrating an example region of interest (ROI) pooling operation implemented in the context of object detection using a convolutional neural network (CNN) according to an example embodiment of the present invention is shown. In various embodiments, the entire input image is typically forwarded through several convolutional layers 250 to generate a convolutional feature map 252. In examples, the feature map typically has a size of C×H×W, where C, H, and W represent the depth (e.g., the number of channels), height, and width of the feature map.

[0067] The convolutional feature map 252 can be used as input to a region proposal generator. In various embodiments, the region proposal generator can be an external proposal method or an internal subnetwork. The region proposal generator can be implemented using conventional and / or proprietary techniques. The region proposal generator is typically configured to output a set of proposals for objects (e.g., regions of interest) within an input image (e.g., shown in the original image space 254). The proposals / ROIs can have different sizes. In various embodiments, the proposals / ROIs received from the region proposal generator are typically projected onto the input image 254 based on predetermined criteria to obtain ground truth regions of interest. An ROI pooling operation is performed on each ground truth proposal / ROI according to embodiments of the present invention to generate fixed-size feature maps 256a-256n for each region proposal. Each of the fixed-size feature maps 256a-256n is typically fed into a subsequent per-ROI subnetwork (e.g., a region-based convolutional neural network or R-CNN) or a process for object classification (e.g., car, pedestrian, or cyclist) and bounding box regression / refinement. An example of a region-based convolutional network approach can be found in R. Girschick, “Fast R-CNN” (IEEE International Conference on Computer Vision (ICCV), 2015), which is incorporated herein by reference in its entirety. In this example, the results of region-of-interest pooling can be used to classify objects in an input image.

[0068] refer to Figure 5, a flow chart illustrating a process 400 of a process during a training or inference phase according to an example embodiment of the present invention is shown. According to an example embodiment of the present invention, the process (or method) 400 generally implements a region of interest (ROI) pooling scheme. In an example embodiment, the process 400 may include step (or state) 402, step (or state) 404, step (or state) 406, step (or state) 408, step (or state) 410, and step (or state) 412. In an example embodiment, steps 402-412 may utilize the ROI pooling technique according to an example embodiment of the present invention to implement an object detection scheme (e.g., using ground truth input during quantization in an object detection process as part of a second stage data range analysis).

[0069] Process 400 generally begins at step 402 and moves to step 404. In step 404, process 400 can use the entire input image to generate a convolutional feature map. In an example, the entire input image can be forwarded through several convolutional layers to generate a convolutional feature map of size C×H×W, where C, H, and W represent the depth (e.g., the number of channels, etc.), height, and width of the feature map. Process 400 then moves to step 406.

[0070] At step 406, process 400 can generate proposals for objects within the input image based on the convolutional feature map. In an example embodiment, a region proposal generator, which can be an external proposal method or an internal subnetwork, can be used to generate a set of proposals / ROIs for objects within the input image. Typically, the proposals / ROIs can have inconsistent sizes. Process 400 then moves to step 408.

[0071] In step 408, process 400 assigns a bounding box (BBOX) to each of the object proposals / ROIs using a default anchor. In various embodiments, the assignment may be based on a predetermined criterion. Then, process 400 moves to step 410. In step 410, process 400 may generate a pooling result for each object proposal using the feature map and the bounding box proposal generated in step 408. In an example embodiment, process 400 may generate a pooling result by cropping and resampling (resizing) the corresponding bounding box. In some embodiments, resampling may be performed by a warp tool. In an example, bilinear interpolation may be used as a warp tool to generate a per-ROI feature map of the target size. However, other warp tools (e.g., nearest neighbor interpolation, bicubic interpolation, etc.) may be used to generate a per-ROI feature map of the target size.

[0072] In an example, the mapped region can be programmed by software into bilinear interpolation hardware, which resizes the features of the region, and the result can then be passed as input to a subsequent subnetwork on the hardware. In the case where the ROI is projected into an area in the ground truth space that is smaller than the target size (e.g., 7×7), the original image data can be used and bilinear interpolation applied to the upsampled features of the region.

[0073] Then, process 400 moves to step 412. In step 412, the pooled results can be used for subsequent processing (e.g., as detected objects). In an example, each fixed-size feature map can be fed into a subsequent per-proposal subnet (e.g., RCNN) or a process for object classification (e.g., car, pedestrian, cyclist, etc.) and bounding box regression / refinement.

[0074] refer to Figure 6 , a flow chart illustrating a process 500 that illustrates a process during a quantization phase according to an example embodiment of the present invention is shown. According to an example embodiment of the present invention, the process (or method) 500 generally implements a quantization phase of a region of interest (ROI) pooling scheme. In an example embodiment, the process 500 may include step (or state) 502, step (or state) 504, step (or state) 506, step (or state) 508, step (or state) 510, step (or state) 512, step (or state) 514, and step (or state) 516. In an example embodiment, steps 502-516 may utilize ROI pooling techniques according to an example embodiment of the present invention to implement an object detection scheme (e.g., using ground truth input during quantization in an object detection process as part of a second stage data range analysis).

[0075] Process 500 generally begins at step 502 and moves to step 504. In step 504, process 500 can perform data range analysis to quantize the input image data. Then, process 500 moves to step 506. In step 506, process 500 can use the entire input image to generate a convolutional feature map. In an example, the entire image can be forwarded through several convolutional layers to generate a convolutional feature map of size C×H×W, where C, H, and W represent the depth (e.g., number of channels, etc.), height, and width of the feature map. Then, process 500 moves to step 508.

[0076] At step 508, process 500 can generate proposals for objects within the input image based on the convolutional feature map. In an example embodiment, a region proposal generator, which can be an external proposal method or an internal subnetwork, can be used to generate a set of proposals / ROIs for objects within the input image. Typically, the proposals / ROIs can have inconsistent sizes. Process 500 then moves to step 510.

[0077] In step 510, process 500 assigns each proposal / ROI to a corresponding ground truth bounding box of the object. In various embodiments, the assignment can be based on predetermined criteria. In an example, the ground truth bounding box can be taken from annotations in a dataset such as VOC, COCO, etc. Annotations are typically labeled by individuals or organizations. In the example pipeline, the ground truth from the corresponding annotations can be read. Process 500 then moves to step 512.

[0078] In step 512, process 500 can generate a pooling result for each object proposal using a feature map and a ground truth bounding box instead of a bounding box proposal generated according to a region proposal network. In an example embodiment, process 500 can generate a pooling result by cropping and resampling (resizing) the corresponding ground truth bounding box. In some embodiments, resampling can be performed by a warp tool. In an example, bilinear interpolation can be used as a warp tool to generate a per-ROI feature map of the target size. However, other warp tools (e.g., nearest neighbor interpolation, bicubic interpolation, etc.) can be used to generate a per-ROI feature map of the target size.

[0079] In an example, the mapped region can be programmed by software into bilinear interpolation hardware, which resizes the features of the region, and the result can then be passed as input to a subsequent subnetwork on the hardware. In the case where the ROI is projected into an area in the ground truth space that is smaller than the target size (e.g., 7×7), the original image data can be used and bilinear interpolation applied to the upsampled features of the region.

[0080] Then, the process 500 moves to step 514, where a second data range analysis is performed on the pooled results. Then, the process 500 moves to step 516. In step 516, the pooled results can be used for subsequent processing (e.g., as detected objects). In an example, each fixed-size feature map can be fed into a subsequent per-proposal subnet (e.g., RCNN) or a process for object classification (e.g., car, pedestrian, cyclist, etc.) and bounding box regression / refinement.

[0081] refer to Figure 7 , shows an illustration of which can be implemented Figure 3FIG2 is a diagram illustrating the context of the training and quantization phases of process 200. In an example, system 600 can be configured (e.g., via software) to perform process 200 on a previously generated neural network. In an example, system 600 can include a computer 602 and a computer-readable storage medium 604. In an example, computer 602 can be implemented as a general-purpose computer system. In an example, computer-readable storage medium 604 can include non-volatile media, including but not limited to magnetic storage media (e.g., a hard drive or HDD), optical storage media (e.g., a compact disk (CD), a digital video disk (DVD), a Blu-ray disk (BD)), a solid-state drive (SSD), flash memory, and / or a network-associated storage system (NAS). In an example, computer 602 and computer-readable storage medium 604 can be coupled together to exchange programs and data. In an example, program(s) implementing process 200 can be stored on computer-readable storage medium 604 or in cloud-based resources 606. In an example, computer 602 can be further configured to utilize cloud resources 606 to perform process 200. In an example, computer 602 may be configured to perform process 200 via an application program interface (API).

[0082] In the example, the computer 602 may include, but is not limited to, a processor 610, a memory 612, a display 614, and a user interface 616. In various embodiments, the processor 610 may include, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), and a video processing unit (VPU). In various embodiments, the memory 612 may include, but is not limited to, a random access memory (e.g., SRAM, DRAM, FLASH, etc.), a read-only memory (ROM), and a cache memory. The display 614 and the user interface 616 generally allow a user to start and monitor the computer 602 executing the process 200 during the training and quantization phases.

[0083] refer to Figure 8 , a diagram of a camera system 900 is shown, which illustrates an example implementation of a computer vision system in which a quantized multi-stage object detection process according to an example embodiment of the present invention may be implemented and / or deployed. In one example, the electronics of the camera system 900 may be implemented as one or more integrated circuits. In an example, the camera system 900 may be built around a processor / camera chip (or circuit) 902. In an example, the processor / camera chip 902 may be implemented as an application specific integrated circuit (ASIC) or a system on a chip (SOC). The processor / camera circuit 902 typically combines hardware and / or software / firmware that may be configured to implement the above combined Figures 1 to 5 Describe the circuit and process.

[0084] In an example, the processor / camera circuit 902 may be connected to a lens and sensor assembly 904. In some embodiments, the lens and sensor assembly 904 may be a component of the processor / camera circuit 902 (e.g., a SoC component). In some embodiments, the lens and sensor assembly 904 may be a separate component from the processor / camera circuit 902 (e.g., the lens and sensor assembly may be an interchangeable component compatible with the processor / camera circuit 902). In some embodiments, the lens and sensor assembly 904 may be part of a separate camera connected to the processor / camera circuit 902 (e.g., via a video cable, a High-Definition Media Interface (HDMI) cable, a Universal Serial Bus (USB) cable, an Ethernet cable, or a wireless link).

[0085] The lens and sensor assembly 904 may include a block (or circuit) 906 and / or a block (or circuit) 908. Circuit 906 may be associated with the lens assembly. Circuit 908 may be implemented as one or more image sensors. In one example, circuit 908 may be implemented as an RGB sensor and / or an IR sensor. In another example, circuit 908 may be implemented as an RGB-IR sensor. The lens and sensor assembly 904 may include other components (not shown). The number, type, and / or function of the components of the lens and sensor assembly 904 may vary according to the design criteria of a particular implementation.

[0086] Lens assembly 906 can capture and / or focus light input received from the environment surrounding camera system 900. Lens assembly 906 can capture and / or focus light for image sensor(s) 908. Lens assembly 906 can implement one or more optical lenses. Lens assembly 906 can provide zoom features and / or focusing features. Lens assembly 906 can be implemented with additional circuitry (e.g., a motor) to adjust the direction, zoom, and / or aperture of lens assembly 906. Lens assembly 906 can be oriented, tilted, translated, zoomed, and / or rotated to provide a desired view of the environment surrounding camera system 900.

[0087] Image sensor 908 can receive light from lens assembly 906. Image sensor 908 can be configured to convert the received focused light into digital data (e.g., a bit stream). In some embodiments, image sensor 908 can perform analog-to-digital conversion. For example, image sensor 908 can perform photoelectric conversion on the focused light received from lens assembly 906. Image sensor 908 can present the converted image data as a bit stream in a color filter array (CFA) format. Processor / camera circuit 902 can convert the bit stream into video data, a video file, and / or a video frame (e.g., human-readable content).

[0088] The processor / camera circuit 902 may also be connected to: (i) optional audio input / output circuitry including an audio codec 910, a microphone 912, and a speaker 914; (ii) memory 916, which may include dynamic random access memory (DRAM); (iii) non-volatile memory (e.g., NAND flash memory) 918, removable media (e.g., SD, SDXC, etc.) 920, one or more serial (e.g., RS-485, RS-232, etc.) devices 922, one or more universal serial bus (USB) devices (e.g., a USB host) 924, and a wireless communication device 926.

[0089] In various embodiments, the processor / camera circuit 902 may include a plurality of blocks (or circuits) 930a-930n, a plurality of blocks (or circuits) 932a-932n, a block (or circuit) 934, a block (or circuit) 936, a block (or circuit) 938, a block (or circuit) 940, a block (or circuit) 942, a block (or circuit) 944, a block (or circuit) 946, a block (or circuit) 948, a block (or circuit) 950, a block (or circuit) 952, and / or a block (or circuit) 954. The plurality of circuits 930a-930n may be processor circuits. In various embodiments, the circuits 930a-930n may include one or more embedded processors (e.g., ARM, etc.). The circuits 932a-932n may implement a plurality of computer vision-related processor circuits. In an example, one or more of the circuits 932a-932n may implement various computer vision-related applications. The circuit 934 may be a digital signal processing (DSP) module. In some embodiments, circuitry 934 may implement separate image DSP and video DSP modules.

[0090] Circuit 936 may be a storage interface. Circuit 936 may interface the processor / camera circuit 902 with the DRAM 916, non-volatile memory 918, and removable media 920. One or more of the DRAM 916, non-volatile memory 918, and / or removable media 920 may store computer-readable instructions. The computer-readable instructions may be read and executed by processors 930a-930n. In response to the computer-readable instructions, processors 930a-930n may be operable to act as controllers for processors 932a-932n. For example, the resources of processors 932a-932n may be configured to efficiently perform various specific operations in hardware, and processors 930a-930n may be configured to make decisions regarding how to handle input / output to / from the various resources of processor 932.

[0091] Circuit 938 may implement a local memory system. In some embodiments, local memory system 938 may include, but is not limited to, a cache memory (e.g., L2CACHE), a direct memory access (DMA) engine, a graphics direct memory access (GDMA) engine, and fast random access memory. In an example, DAG memory 98 may be implemented in local memory system 938. Circuit 940 may implement sensor inputs (or interfaces). Circuit 942 may implement one or more control interfaces, including, but not limited to, an inter-device communication (IDC) interface, an inter-integrated circuit (I2C) interface, a serial peripheral interface (SPI), and a pulse width modulation (PWM) interface. Circuit 944 may implement an audio interface (e.g., an I2S interface, etc.). Circuit 946 may implement clock circuits, including, but not limited to, a real-time clock (RTC), a watchdog timer (WDT), and / or one or more programmable timers. Circuit 948 may implement an input / output (I / O) interface. Circuit 950 may be a video output module. Circuit 952 may be a communications module. Circuit 954 may be a security module. Circuits 930 - 954 may be connected to each other using one or more buses, interfaces, traces, protocols, or the like.

[0092] Circuit 918 may be implemented as a non-volatile memory (e.g., NAND flash memory, NOR flash memory, etc.). Circuit 920 may include one or more removable media cards (e.g., Secure Digital (SD), Secure Digital Extended Capacity (SDXC), etc.). Circuit 922 may include one or more serial interfaces (e.g., RS-485, RS-232, etc.). Circuit 924 may be an interface for connecting to a Universal Serial Bus (USB) host or acting as a USB host. Circuit 926 may be a wireless interface for communicating with a user device (e.g., a smartphone, a computer, a tablet computing device, a cloud resource, etc.). In various embodiments, circuits 904-926 may be implemented as components external to processor / camera circuit 902. In some embodiments, circuits 904-926 may be on-board components of processor / camera circuit 902.

[0093] The control interface 942 may be configured to generate signals (e.g., IDC / I2C, STEPPER, IRIS, AF / ZOOM / TILT / PAN, etc.) for controlling the lens and sensor assembly 904. The signal IRIS may be configured to adjust the aperture of the lens assembly 906. The interface 942 may enable the processor / camera circuit 902 to control the lens and sensor assembly 904.

[0094] The storage interface 936 can be configured to manage one or more types of storage and / or data access. In one example, the storage interface 936 can implement a direct memory access (DMA) engine and / or graphics direct memory access (GDMA). In another example, the storage interface 936 can implement a secure digital (SD) card interface (e.g., to connect to the removable media 920). In various embodiments, programming code (e.g., executable instructions for controlling the various processors and encoders of the processor / camera circuit 902) can be stored in one or more of the memories (e.g., DRAM 916, NAND 918, etc.). When executed by one or more of the processors 930, the programming code typically causes one or more components in the processor / camera circuit 902 to configure video synchronization operations and initiate video frame processing operations. The resulting compressed video signal can be presented to the storage interface 936, the video output 950, and / or the communication interface 952. The storage interface 936 can transfer program code and / or data between external media (e.g., DRAM 916, NAND 918, removable media 920, etc.) and the local (internal) memory system 938.

[0095] The sensor input 940 can be configured to send data to / receive data from the image sensor 908. In one example, the sensor input 940 can include an image sensor input interface. The sensor input 940 can be configured to send a captured image (e.g., picture elements, pixels, data) from the image sensor 908 to the DSP module 934, one or more of the processors 930, and / or one or more of the processors 932. The data received by the sensor input 940 can be used by the DSP 934 to determine luma (Y) and chroma (U and V) values ​​from the image sensor 908. The sensor input 940 can provide an interface to the lens and sensor assembly 904. The sensor input interface 940 can enable the processor / camera circuit 902 to capture image data from the lens and sensor assembly 904.

[0096] The audio interface 944 may be configured to transmit / receive audio data. In one example, the audio interface 944 may implement an audio inter-IC sound (I2S) interface. The audio interface 944 may be configured to transmit / receive data in a format implemented by the audio codec 910.

[0097] The DSP module 934 can be configured to process digital signals. The DSP module 934 can include an image digital signal processor (IDSP), a video digital signal processor DSP (VDSP), and / or an audio digital signal processor (ADSP). The DSP module 934 can be configured to receive information from the sensor input 940 (e.g., pixel data values ​​captured by the image sensor 908). The DSP module 934 can be configured to determine pixel values ​​(e.g., RGB, YUV, luminance, chrominance, etc.) based on the information received from the sensor input 940. The DSP module 934 can be further configured to support or provide a sensor RGB to YUV raw image pipeline to improve image quality, perform bad pixel detection and correction, demosaicing, white balance, color and tone correction, gamma correction, hue adjustment, saturation, brightness and contrast adjustment, and chroma and luminance noise filtering.

[0098] The I / O interface 948 may be configured to send / receive data. The data sent / received by the I / O interface 948 may be miscellaneous information and / or control data. In one example, the I / O interface 948 may implement one or more of a general-purpose input / output (GPIO) interface, an analog-to-digital converter (ADC) module, a digital-to-analog converter (DAC) module, an infrared (IR) remote interface, a pulse width modulation (PWM) module, a universal asynchronous receiver transmitter (UART), an infrared (IR) remote interface, and / or one or more synchronous data communication interfaces (IDC SPI / SSI).

[0099] The video output module 950 can be configured to transmit video data. For example, the processor / camera circuit 902 can be connected to an external device (e.g., a television, a monitor, a laptop computer, a tablet computing device, etc.). The video output module 950 can implement a High-Definition Multimedia Interface (HDMI), a PAL / NTSC interface, an LCD / TV / parallel interface, and / or a DisplayPort interface.

[0100] The communication module 952 may be configured to send / receive data. The data sent / received by the communication module 952 may be transmitted / received according to a specific protocol (e.g., In one example, the communication module 952 may implement a secure digital input output (SDIO) interface. The communication module 952 may include a communication interface that receives data transmitted via one or more wireless protocols (e.g., The processor / camera circuit 902 may also include support for wireless communications using Z-Wave, LoRa, Institute of Electrical and Electronics Engineers (IEEE) 802.11a / b / g / n / ac (WiFi), IEEE 802.15, IEEE 802.15.1, IEEE 802.15.2, IEEE 802.15.3, IEEE 802.15.4, IEEE 802.15.5, and / or IEEE 802.20, GSM, CDMA, GPRS, UMTS, CDMA2000, 3GPP LTE, 4G / HSPA / WiMAX, 5G, LTE-M, NB-IoT, SMS, etc. The communication module 952 may also include support for communication using one or more universal serial bus protocols (e.g., USB 1.0, 2.0, 3.0, etc.). The processor / camera circuit 902 may also be configured to be powered via a USB connection. However, other communication and / or power interfaces may be implemented accordingly to meet the design criteria of a particular application.

[0101] The security module 954 may include a set of advanced security features to achieve advanced on-device physical security, including OTP, secure boot, As well as I / O visualization and DRAM scrambling. In an example, the security module 958 can include a true random number generator. In an example, the security module 954 can be used for DRAM communication encryption on the processor / camera circuit 902.

[0102] The processor / camera circuit 902 can be configured (e.g., programmed) to control one or more lens assemblies 906 and one or more image sensors 908. The processor / camera circuit 902 can receive raw image data from the image sensor(s) 908. The processor / camera circuit 902 can simultaneously (in parallel) encode the raw image data into multiple encoded video streams. The multiple video streams can have various resolutions (e.g., VGA, WVGA, QVGA, SD, HD, Ultra HD, 4K, etc.). The processor / camera circuit 902 can receive encoded and / or unencoded (e.g., raw) audio data at the audio interface 944. The processor / camera circuit 902 can also receive encoded audio data from the communication interface 952 (e.g., USB and / or SDIO). The processor / camera circuit 902 can provide the encoded video data to the wireless interface 926 (e.g., using a USB host interface). The wireless interface 926 can include support for video signals transmitted via one or more wireless and / or cellular protocols (e.g., The processor / camera circuit 902 may also include support for wireless communications using Z-Wave, LoRa, Wi-Fi (IEEE 802.11a / b / g / n / ac, IEEE 802.15, IEEE 802.15.1, IEEE 802.15.2, IEEE 802.15.3, IEEE 802.15.4, IEEE 802.15.5, IEEE 802.20, GSM, CDMA, GPRS, UMTS, CDMA2000, 3GPP LTE, 4G / HSPA / WiMAX, 5G, SMS, LTE-M, NB-IoT, etc. The processor / camera circuit 902 may also include support for communication using one or more universal serial bus protocols (e.g., USB 1.0, 2.0, 3.0, etc.).

[0103] As will be apparent to those skilled in the relevant art(s), Figures 1 to 8 The functions and structures shown in the figures can be designed, modeled, simulated and / or emulated using one or more conventional general-purpose processors, digital computers, microprocessors, microcontrollers, distributed computer resources and / or similar computing machines, or programmed according to the teachings of this specification. As will also be apparent to those skilled in the relevant art(s), skilled programmers can readily prepare appropriate software, firmware, coding, routines, instructions, opcodes, microcode and / or program modules based on the teachings of this disclosure. The software is typically embodied in one or more media (e.g., non-transitory storage media) and can be executed sequentially or in parallel by one or more of the processors.

[0104] Embodiments of the present invention may also be implemented in one or more of an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), a PLD (programmable logic device), a CPLD (complex programmable logic device), an ocean of gates, an ASSP (application specific standard product), and an integrated circuit. The circuit may be implemented based on one or more hardware description languages. Embodiments of the present invention may be utilized in conjunction with flash memory, nonvolatile memory, random access memory, read-only memory, magnetic disks, floppy disks, optical disks (e.g., DVDs and DVDRAMs), magneto-optical disks, and / or distributed storage systems.

[0105] When used herein in conjunction with the verb "is," the terms "may" and "generally" are intended to convey the intention that the description is exemplary and is to be considered broad enough to encompass both the specific examples presented in this disclosure and alternative examples that can be derived based on this disclosure. The terms "may" and "generally" as used herein should not be interpreted as necessarily implying the desirability or possibility of omitting the corresponding element.

[0106] While the invention has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the invention.

Claims

1. A device comprising: a memory configured to store image data of an input image; as well as A processor configured to detect one or more objects in the input image using a quantized multi-stage object detection network, wherein quantizing the quantized multi-stage object detection network includes: (i) generating quantized image data by performing a first data range analysis on the image data of the input image; (ii) generating a feature map and a proposal bounding box by applying a region proposal network to the quantized image data; (iii) selecting a plurality of ground truth bounding boxes from the input image; (iv) performing a region of interest pooling operation using the feature map and the plurality of ground truth bounding boxes of the input image instead of the proposal bounding boxes generated by the region proposal network; (v) generating a quantized region of interest pooling result by performing a second data range analysis on a result from the region of interest pooling operation, wherein the region of interest pooling operation is performed using the feature map and the plurality of ground truth bounding boxes of the input image; and (vi) applying a region-based convolutional neural network to the quantized region of interest pooling result.

2. The device according to claim 1, wherein: The ground truth bounding box is the projection of the proposal bounding box on the input image.

3. The device according to claim 1, wherein: The second data range analysis applies the technique applied by the first data range analysis.

4. The device according to claim 1, wherein: Training the region proposal network and the region-based convolutional neural network includes: (i) generating a feature map and a proposal bounding box by applying the region proposal network to the image data of the input image; and (ii) performing the region of interest pooling operation on the feature map and the proposal bounding box generated by the region proposal network.

5. The device according to claim 1, wherein: The region proposal network and the region-based convolutional neural network are stored in the processor as directed acyclic graphs and corresponding weights.

6. The device according to claim 1, wherein: The region proposal network and the region-based convolutional neural network share one or more convolutional layers.

7. The device according to claim 1, wherein: The processor is further configured to generate a pooling result for the region of interest by cropping and resampling a corresponding portion of the feature map to which the object detection proposal is assigned.

8. The device according to claim 7, wherein: The resampling includes a warping operation.

9. The device according to claim 1, wherein: The processor configures bilinear interpolation hardware to resize features of the region of interest for use as input to a subsequent subnetwork in hardware.

10. The device according to claim 1, wherein: The feature map includes a three-dimensional array, and the dimensions of the three-dimensional array correspond to the depth, height and width of the feature map.

11. The device according to claim 1, wherein: The memory and the processor are part of at least one of a computer vision system or an autonomous vehicle.

12. A method for object detection, comprising: storing image data of an input image in a memory; as well as A processor is used to execute a quantized multi-stage object detection network to detect one or more objects in the input image, wherein the quantization of the quantized multi-stage object detection network is performed by: (i) generating quantized image data by performing a first data range analysis on the image data of the input image; (ii) generating a feature map and a proposal bounding box by applying a region proposal network to the quantized image data; (iii) selecting a plurality of ground truth bounding boxes from the input image; (iv) performing a region of interest pooling operation using the feature map and the plurality of ground truth bounding boxes of the input image instead of the proposal bounding boxes generated by the region proposal network; (v) generating a quantized region of interest pooling result by performing a second data range analysis on a result from the region of interest pooling operation, wherein the region of interest pooling operation is performed using the feature map and the plurality of ground truth bounding boxes of the input image; and (vi) applying a region-based convolutional neural network to the quantized region of interest pooling result.

13. The method according to claim 12, wherein: The ground truth bounding box is the projection of the proposal bounding box on the input image.

14. The method according to claim 12, further comprising training the region proposal network and the region-based convolutional neural network, wherein: Training the region proposal network and the region-based convolutional neural network includes: (i) generating a feature map and a proposal bounding box by applying the region proposal network to the image data of the input image; and (ii) performing the region of interest pooling operation on the feature map and the proposal bounding box generated by the region proposal network.

15. The method according to claim 12, wherein: The region proposal network and the region-based convolutional neural network are stored in the processor as directed acyclic graphs and corresponding weights.

16. The method according to claim 12, wherein: The region proposal network and the region-based convolutional neural network share one or more convolutional layers.

17. The method according to claim 12, further comprising: A pooling result for the region of interest is generated by performing cropping and resampling, using the processor, on a corresponding portion of the feature map to which the object detection proposal is assigned.

18. The method according to claim 17, wherein: Resampling involves applying a warp operation.

19. The method of claim 12, further comprising using the processor to configure bilinear interpolation hardware to resize features of a region of interest for use as input to a subsequent subnetwork in hardware.

20. The method according to claim 12, wherein: The feature map includes a three-dimensional array, and the dimensions of the three-dimensional array correspond to the depth, height and width of the feature map.

Images