Object detection method and object detection system

Abstract

The invention provides a method, equipment and a system for object detection. The present implementations more particularly relate to techniques for improving remote object detection in memory constrained computer vision systems. In some aspects, a computer vision system may include an ROI extraction component, a feature pyramid network (FPN) having a number of N pyramid stages, and N network headers associated with the N pyramid stages, respectively. The FPN extracts N feature maps from the input image, where the N feature maps are respectively associated with the N pyramid levels, and each of the N network headers performs an object detection operation on a respective one of the N feature maps. The ROI extraction section selects a region of the feature map associated with the lowest pyramid level to perform remote object detection such that an object detection operation performed on the feature map is limited to the selected region.

Description

Technical Field

[0001] The present implementations relate generally to object detection, and in particular to an object detection network for long-range object detection in memory-constrained devices. Background Art

[0002] Computer vision is the field of artificial intelligence (AI) that simulates the human visual system to draw inferences about an environment from an image or video of the environment. Example computer vision techniques include object detection, object classification, and object tracking, among other examples. Object detection includes various techniques for detecting objects in an environment that belong to known categories, such as humans, cars, or text. For example, the presence and location of an object can be detected or inferred by scanning an image for a unique set of features, such as eyes, noses, and lips, for a particular class of objects, such as humans. Some object detection techniques rely on statistical models for feature extraction, while other object detection techniques rely on neural network models for feature extraction. Such models can be used to localize objects in an image, and can generally be referred to as "object detection models."

[0003] The memory and processing resources required for object detection generally grow in proportion to the distance of the object to be detected. For example, objects farther away from an image capture device (such as a camera) appear smaller in the captured image, with fewer or less obvious features. As such, high-resolution object detection models are typically required to detect distant objects in an image. However, many high-resolution object detection models require a large amount of memory and processing resources to achieve accurate detection results. Since computer vision is typically implemented by edge devices with limited resources (such as battery-powered cameras), there is a need to improve the accuracy of distant object detection without increasing the memory budget of the object detection model. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] An innovative aspect of the subject matter of the present disclosure can be implemented in a method for object detection. The method includes the following steps: receiving an input image; extracting multiple feature maps from the input image based on a feature pyramid network (FPN) having multiple pyramid levels, each feature map in the multiple feature maps being associated with a corresponding pyramid level in the multiple pyramid levels; selecting an area of ​​a first feature map in the multiple feature maps for long-range object detection; and performing an object detection operation on each feature map in the multiple feature maps based on a corresponding network head, so that the object detection operation performed on the first feature map is limited to the selected area.

[0006] Another innovative aspect of the subject matter of the present disclosure can be implemented in a method for object detection. The method includes the following steps: receiving an input image; selecting an area of ​​the input image for long-distance object detection; extracting a first feature map from a plurality of feature maps from the selected area of ​​the input image based on a backbone convolutional neural network (CNN); extracting one or more second feature maps from the plurality of feature maps from the input image based on an FPN having a plurality of pyramid levels, each feature map in the plurality of feature maps being associated with a corresponding pyramid level in the plurality of pyramid levels; and performing an object detection operation on each feature map in the plurality of feature maps based on a corresponding network head.

[0007] Another innovative aspect of the subject matter of the present disclosure can be implemented in an object detection system, which includes a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the object detection system to: receive an input image; select an area of ​​the input image for long-range object detection; extract a first feature map from a plurality of feature maps from the selected area of ​​the input image based on a backbone CNN; extract one or more second feature maps from the plurality of feature maps based on an FPN having a plurality of pyramid levels, each feature map in the plurality of feature maps being associated with a corresponding pyramid level in the plurality of pyramid levels; and perform an object detection operation on each feature map in the plurality of feature maps based on a corresponding network head. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present embodiments are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.

[0009] Figure 1 A block diagram of an example computer vision system is shown in accordance with some implementations.

[0010] Figure 2A Example neural network operations that can be performed by a convolutional neural network (CNN) are shown.

[0011] Figure 2B Another example neural network operation that can be performed by a CNN is shown.

[0012] Figure 3 Block diagram showing an example object detection network based on a Feature Pyramid Network architecture.

[0013] Figure 4 An example image that can be captured by a computer vision system is shown.

[0014] Figure 5 A block diagram illustrating an example object detection network according to some implementations.

[0015] Figure 6Example regions of interest (ROIs) that can be extracted from example feature maps are shown.

[0016] Figure 7 A block diagram illustrating another example object detection network in accordance with some implementations.

[0017] Figure 8 An example ROI that can be extracted from an example input image is shown.

[0018] Fig. 9 A block diagram of an example object detection system is shown in accordance with some implementations.

[0019] Fig.10 A block diagram of another example object detection system is shown in accordance with some implementations.

[0020] Fig.11 An illustrative flow diagram depicting example object detection operations in accordance with some implementations is shown.

[0021] Fig.12 An illustrative flow chart depicting another example object detection operation in accordance with some implementations is shown. DETAILED DESCRIPTION

[0022] In the following description, many specific details, such as examples of specific components, circuits and processes, are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" represents being directly connected to or connected through one or more intermediate components or circuits. The terms "electronic system" and "electronic device" can be used interchangeably to refer to any system capable of electronically processing information. Moreover, in the following description, and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to practice the exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid making the present disclosure difficult to understand. The subsequent parts of the detailed description are presented in accordance with other symbolic representations of procedures, logic blocks, processing and the operation of data bits in a computer memory.

[0023] These descriptions and representations are the means by which the technical personnel in the data processing field use to most effectively communicate the essence of their work to other technical personnel in the field. In the present disclosure, procedures, logic blocks, processes, etc. are considered to be the self-consistent sequence of steps or instructions that lead to a desired result. Steps are those steps that require the physical operation of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise operated in a computer system. However, it should be remembered that all of these terms and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities.

[0024] Unless otherwise specifically stated, as will be apparent from the following discussion, it is appreciated that throughout this application, discussions utilizing terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," and the like refer to the actions and processes of a computer system or similar electronic computing device to manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.

[0025] In the accompanying drawings, a single block may be described as implementing one or more functions; however, in actual practice, one or more functions implemented by the block may be implemented in a single component or across multiple components, and / or may be implemented using hardware, using software, or using a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described below in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as departing from the scope of the present invention. Moreover, the example input device may include components different from those shown, including known components such as processors, memories, etc.

[0026] Unless the techniques described herein are specifically described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium that includes instructions that, when executed, implement one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0027] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the technology may be at least partially implemented by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0028] The various illustrative logical blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein may be executed by one or more processors (or processing systems). The term "processor" as used herein may refer to any general purpose processor, special purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0029] As described above, object detection includes various techniques for detecting objects in images belonging to known categories. The memory and processing resources required for object detection generally grow in proportion to the distance of the object to be detected. Specifically, high-resolution object detection models are generally required to detect distant objects in images. However, computer vision is typically implemented in edge devices (such as battery-powered cameras) with limited memory and processing resources. Aspects of the present disclosure recognize that such edge devices are typically mounted to a fixed location (such as a wall, television, or computer display) where distant objects are confined to a small region of interest (ROI) within the captured image. As such, by focusing a high-resolution object detection model on the ROI in which distant objects are confined, the accuracy of distant object detection can be improved without increasing the memory budget.

[0030] Various aspects generally relate to object detection and, more particularly, to improving the detection of distant objects by a memory-constrained computer vision system. In some aspects, the computer vision system can include a ROI extraction component, a Feature Pyramid Network (FPN) having several (N) pyramid levels, and N network heads respectively associated with the N pyramid levels. In such aspects, the FPN is configured to extract N feature maps from an input image, where the N feature maps are respectively associated with the N pyramid levels, and each of the N network heads is configured to perform object detection operations on a corresponding one of the N feature maps. In some implementations, the ROI extraction component can select regions of the feature map associated with the lowest pyramid level among the N pyramid levels for distant object detection. For example, the selected regions can coincide with the ROI of the input image within which the distant object is confined. As a result, the object detection operations (performed by the corresponding network heads) on the feature map associated with the lowest pyramid level are confined to the selected regions.

[0031] In some other aspects, the computer vision system can include a ROI extraction component, a distant object backbone network, an FPN having several (N) pyramid levels, and N network heads respectively associated with the N pyramid levels. In some implementations, the ROI extraction component can select regions of the input image for distant object detection. For example, the selected regions can represent the ROI of the input image within which the distant object is confined. The distant object backbone network can extract several (M) feature maps from the selected regions of the input image, where 1 ≤ M < N, and the M feature maps are associated with M of the lowest pyramid levels among the N pyramid levels. The FPN is configured to extract an additional N - M feature maps from the input image (as a whole), where the N - M feature maps are associated with the N - M remaining pyramid levels. Each of the N network heads is configured to perform object detection operations on a corresponding one of the N feature maps (which includes the first feature maps extracted by the distant object backbone network and the N - M feature maps extracted by the FPN).

[0032] Specific implementations of the subject matter described in the present disclosure can be implemented to achieve one or more of the following potential advantages. As described above, distant objects in an input image are typically detected by a high-resolution object detection model associated with the lowest pyramid level of the FPN. By focusing the high-resolution object detection model on an ROI in which distant objects are confined, aspects of the present disclosure can improve the accuracy of distant object detection in memory-constrained edge devices. For example, by reducing the size of feature maps to be processed by a network head associated with the lowest pyramid level, aspects of the present disclosure can increase the number of neural network filters in a network head compared to existing computer vision systems with the same memory budget. Moreover, by reducing the size of input images to be processed by a backbone network associated with the M lowest pyramid levels, aspects of the present disclosure can increase the number of neural network filters in a network backbone compared to existing computer vision systems with the same memory budget.

[0033] Figure 1 A block diagram of an example computer vision system 100 is shown according to some implementations. In some aspects, the computer vision system 100 can be configured to generate inferences about one or more objects of interest (also referred to as "target objects"). Figure 1 In the example of , the object of interest 101 is depicted as a person. In some other implementations, the computer vision system 100 can be configured to generate inferences about various other objects of interest in addition to or instead of the object of interest 101 .

[0034] The system 100 includes an image capture component 110 and an image analysis component 120. The image capture component 110 may be any sensor or device (such as a camera) configured to capture a light pattern in its field of view (FOV) 112 and convert the light pattern into a digital image 102. For example, the digital image 102 may include an array of pixels (or pixel values) representing the light pattern in the FOV 112 of the image capture component 110. In some implementations, the image capture component 110 may continuously (or periodically) capture a series of images 102 representing a digital video. Figure 1 As shown in FIG. 1 , the object of interest 101 is located within the FOV 112 of the image capture component 110 . As a result, the digital image 102 may include or depict the object of interest 101 .

[0035] The image analysis component 120 is configured to generate one or more inferences 103 based on the digital image 102. In some aspects, the image analysis component 120 can infer whether one or more objects of interest 101 are depicted in the image 102. For example, the image analysis component 120 can detect a person in the digital image 102 and draw a bounding box around the person's face. In other words, the image analysis component 120 can generate an annotated image as an inference 103, indicating the location of the object of interest 101 relative to the image 102. In some aspects, the location of the object of interest 101 can change over time, for example, based on the movement of the object of interest 101. Therefore, the image analysis component 120 can generate different inferences 103 in response to images 102 captured at different times.

[0036] In some aspects, image analysis component 120 can generate inferences 103 based on object detection model 122. Object detection model 122 can be trained or otherwise configured to detect objects in an image or video. For example, object detection model 122 can apply one or more transformations to pixels in image 102 to create one or more features that can be used for object detection. More specifically, object detection model 122 can compare features extracted from image 102 with a known set of features that uniquely identify a particular class of objects (such as humans) to determine the presence or location of any target object in image 102. In some implementations, object detection model 122 can be a machine learning model.

[0037] Machine learning, which generally includes a training phase and an inference phase, is a technique for improving the ability of a computer system or application to perform specific tasks. During the training phase, a machine learning system is provided with multiple "answers" and a large amount of raw input data. For example, the input data may include an image depicting an object of interest, and the answer may include a bounding box indicating the presence and location of the object of interest. The machine learning system analyzes the input data to "learn" a set of rules (also referred to as a machine learning "model") that can be used to map the input data to the answer. During the inference phase, the machine learning system uses the trained model to infer answers (such as bounding boxes) from new input data.

[0038] Deep learning is a specific form of machine learning in which the model being trained is a multi-layer neural network. Due to the way in which information is processed (similar to biological nervous systems), deep learning architectures are often referred to as artificial neural networks. For example, each layer of a deep learning architecture is formed by several artificial neurons. Neurons are interconnected across various layers so that input data can be passed from one layer to another. More specifically, each layer of neurons can perform a different set of transformations on the input data (or output from the previous layer), which ultimately leads to the desired inference.

[0039] A convolutional neural network (CNN) is a specific type of artificial neural network that processes data in a manner similar to the human visual system. As such, CNNs are well suited for computer vision applications. Each neuron in a CNN responds to a corresponding subset of data from the previous layer that is within its "receptive field" by outputting an "activation" that represents a higher-level abstraction of the features in its receptive field. The receptive fields of neurons in a given layer are combined to cover the entire input from the previous layer (similar to how the receptive fields of cortical neurons in the brain cover the entire visual field). The set of activations output by each layer is collectively referred to as a "feature map."

[0040] Figure 2A 200 is an example neural network operation that can be performed by a CNN. More specifically, the CNN can generate a feature map 206 by applying a filter 204 (also referred to as a "neural network filter") to the pixel values ​​of an input image 202. For example, referring to Figure 1 , input image 202 may be an example of image 102. In some aspects, filter 204 may be a convolution filter associated with a convolutional layer of a CNN.

[0041] exist Figure 2A In the example of FIG. 1 , the input image 202 is depicted as pixel values ​​a 1,1 -a 8,8 , the filter 204 is depicted as a 3x3 matrix with filter weights b1-b9, and the feature map 206 is depicted as an activation c 1,1 -c 8,8 8x8 array. However, in actual implementations, the input image 202, the filter 204, and the feature map 206 may have any suitable size. The feature map 206 is generated by applying the filter 204 to various 3x3 subarrays of pixel values ​​in the input image 202 with a stride of 1 and the same padding (which adds zero values ​​to the borders of the input image 202 to participate in the convolution).

[0042] For example, if Figure 2A As shown in the shaded area, the CNN may apply filter 204 to pixel value a 2,2 、a 2,3 、a 2,4 、a 3,2 、a 3,3 、a 3,4 、a 4,2 、a 4,3 and a 4,4 to generate activation c 3,3 (where c 3,3 =b1·a 2,2 +b2·a 2,3 +b3·a 2,4 +b4·a3,2 +b5·a 3,3 +b6·a 3,4 +b7·a 4,2 +b8·a 4,3 +b9·a 4,4 ). Therefore, the pixel value a 2,2 、a 2,3 、a 2,4 、a 3,2 、a 3,3 、a 3,4 、a 4,2 、a 4,3 and a 4,4 Falling into the activation 3,3 The remaining activations of the feature map 206 can be generated in a similar manner by sliding the filter 204 over the input image 202.

[0043] Figure 2B Another example neural network operation 210 that can be performed by a CNN is shown. More specifically, the CNN can generate another feature map 214 by applying a filter 212 (also referred to as a "neural network filter") to the activations of feature map 206. For example, feature map 214 can be associated with a deeper CNN layer than feature map 206. In some aspects, filter 212 can be a pooling filter associated with a pooling layer of a neural network.

[0044] exist Figure 2B In the example of FIG. 2 , the filter 212 is depicted as a 3x3 pooling matrix, and the feature map 214 is depicted as an activation d 1,1 -d 6,6 6x6 array. However, in actual implementations, filter 212 and feature map 214 may have any suitable size. Feature map 214 may be generated by applying filter 212 to various 3x3 subarrays of activations in feature map 206 with a stride of 1 and effective padding (or no padding).

[0045] For example, if Figure 2B As shown in the shaded area, the neural network can apply filter 212 to the activation c 2,2 、c 2,3 、c 2,4 、c 3,2 、c 3,3 、c 3,4 、c 4,2 、c 4,3 and c 4,4 to generate activation d 2,2 Therefore, the pixel value c 2,2 、c 2,3 、c 2,4 、c3,2 、c 3,3 、c 3,4 、c 4,2 、c 4,3 and c 4,4 Falling into the activation 2,2 In some aspects, filter 212 can be configured to perform a maximum pooling operation (where d 2,2 =max(c 2,2 、c 2,3 、c 2,4 、c 3,2 、c 3,3 、c 3,4 、c 4,2 、c 4,3 、c 4,4 )). In some other aspects, the filter 312 can be configured to perform an average pooling operation (where d 2,2 =(c 2,2 +c 2,3 +c 2,4 +c 3,2 +c 3,3 +c 3,4 +c 4,2 +c 4,3 +c 4,4 ) / 9). The remaining activations of feature map 214 can be generated in a similar manner by sliding filter 212 on feature map 206.

[0046] like Figure 2A and Figure 2B As shown in , CNN extracts features from an input image in a hierarchical manner. For example, neurons associated with lower (or earlier) layers of the CNN are capable of extracting smaller features associated with the input image at a lower level of abstraction. In contrast, neurons associated with higher (or later) layers of the CNN are capable of extracting larger features associated with the input image at a higher level of abstraction. As a result, feature maps generated by lower layers of the CNN tend to have higher spatial resolution but lower semantic value than feature maps generated by higher layers of the CNN. Since distant objects appear small in the input image and have fewer or less obvious features, such objects are more likely to be detected based on features extracted by lower layers of the CNN.

[0047] Figure 3 A block diagram of an example object detection network 300 based on a feature pyramid network architecture is shown. The object detection network 300 is configured to receive an input image 301 and generate a number of inferences 302-305 based on the input image 301. In some implementations, the object detection network 300 may be Figure 1For example, refer to the image analysis unit 120. Figure 1 , input image 301 may be an example of image 102 , and inferences 302 - 305 may be examples of inferences 103 .

[0048] The object detection network 300 includes a feature pyramid network (FPN) 310 and a number of network heads 322-325. The FPN 310 is configured to extract a number of feature maps P2-P5 from an input image 301 based at least in part on a convolutional neural network (CNN). More specifically, each of the feature maps P2-P5 is associated with a corresponding "pyramid level" of the FPN 310. Figure 3 In the example of FIG. 3 , FPN 310 is shown as including 4 pyramid levels. In an actual implementation, FPN 310 may include more than Figure 3 FPN 310 includes a bottom-up path 312 and a top-down path 314.

[0049] The bottom-up path 312 is a feed-forward CNN backbone that generates several intermediate feature maps C2-C5 at various network "levels". A network level is a collection of CNN layers that generate feature maps of the same size or dimension (or number of activations). For example, Figure 2A and Figure 2B , feature map 206 and feature map 214 are associated with different network levels. Each of the intermediate feature maps C2-C5 represents a feature map output by the last layer of the corresponding network level. Therefore, the bottom-up path 312 generates intermediate feature maps C2-C5 in the order of increasing semantic value.

[0050] The top-down path 314 generates feature map P5 by applying a convolution filter to the intermediate feature map C5 to reduce its channel depth. The top-down path 314 generates the remaining feature maps P2-P4 by progressively upsampling the feature map P5 and combining each upsampled feature map with the corresponding intermediate feature map from the bottom-up path 312. For example, the top-down path 314 generates feature map P4 by upsampling the spatial resolution of feature map P5 (to match the resolution of the intermediate feature map C4), applying a convolution filter to the intermediate feature map C4, and merging the result of the convolution with the upsampled feature map (via element-wise addition).

[0051] The top-down path 314 generates feature map P3 by upsampling the spatial resolution of feature map P4 (to match the resolution of intermediate feature map C3), applying a convolution filter to the intermediate feature map C3, and merging the result of the convolution with the upsampled feature map. This process is repeated to generate feature map P2. Thus, the top-down path 314 generates feature maps P2-P5 in order of increasing spatial resolution. In some implementations, additional (3x3) convolution filters may be applied to each of feature maps P2-P5 to reduce aliasing effects due to upsampling.

[0052] exist Figure 3 In the example of , FPN 310 does not produce a feature map (such as P1) that matches the resolution of the intermediate feature map C1 associated with the first network stage of the bottom-up path 312 because of its large memory footprint (and because any objects detected at that level will be too small to be reliably detected without additional contextual information). In some implementations, the top-down path 314 can be omitted from FPN 310. In such an implementation, FPN 310 can include only the bottom-up path 312, and can directly output the intermediate feature maps C2-C5 as feature maps P2-P5, respectively.

[0053] Network heads 322-325 are configured to perform object detection operations on feature maps P2-P5, respectively. As such, each of network heads 322-325 is associated with a corresponding pyramid level of FPN 310. More specifically, network heads associated with lower pyramid levels (such as network head 322) can detect objects based on smaller or finer features. In contrast, network heads associated with higher pyramid levels (such as network head 325) can detect objects based on larger or coarser features.

[0054] In some aspects, each of the network heads 322-325 can be configured to infer a bounding box associated with an object of interest in the input image 301. For example, each network head can detect an object of interest in the input image 301 based on features or activations in its corresponding feature map. Each network head can further map a set of boxes to its corresponding feature map and adjust the box to coincide with any object of interest detected based on the feature map. In some implementations, the set of boxes can include one or more anchor boxes.

[0055] In some implementations, each of the network heads 322-325 may include a classification subnetwork and a box regression subnetwork (not shown for simplicity). For example, the classification subnetwork may infer or indicate the probability of detecting an object of interest at each box location, and the box regression subnetwork may infer or regress the offset of each box relative to a nearby object of interest (if detected). Thus, the accuracy of the inferences 302-305 may depend on the number of neural network filters implemented by the network heads 322-325, respectively.

[0056] Due to the size and limited resolution of distant objects in input image 301, distant objects in input image 301 are more likely to be detected by network head 322 (also referred to as P2 network head) associated with the second pyramid level of FPN 310 than by any of network heads 323-325 associated with higher pyramid levels. However, due to memory constraints of computer vision systems and the high spatial resolution of feature maps P2 associated with the second pyramid level, the number of filters implemented by network head 322 is typically limited. As a result, many existing object detection networks cannot detect objects that are more than a 100% distance from an image capture device (such as a 400% object detection network). Figure 1 of the image capture component 110) to an object at a threshold distance.

[0057] Aspects of the present disclosure recognize that computer vision systems are typically mounted to a fixed location (such as a wall, television, or computer display) where distant objects are confined to a small region of interest (ROI) in the captured image. Thus, in some aspects, by focusing the network head 322 on the ROI in which the distant objects are confined, the accuracy of distant object detection can be improved without increasing the memory budget of the object detection system 300. In some implementations, the ROI can be used to reduce the size (or number of activations) of the feature map P2 (or one or more of the feature maps P3-P5 to be processed by the network heads 323-325, respectively) to be processed by the network head 322. In some other implementations, the ROI can be used to reduce the size (or number of pixels) of the input image 301 to be processed by the backbone CNN that produces the feature map P2 (or one or more of the other feature maps P3-P5).

[0058] Figure 4 An example image 400 that can be captured by a computer vision system is shown. In some implementations, the computer vision system may be Figure 1 An example of a computer vision system 100 is shown in FIG. Figure 1 , image 400 may be an example of image 102 captured by image capture component 110 .

[0059] exist Figure 4In the example of , the image capture component 110 is mounted to a wall at the end of a hallway, and the FOV 112 of the image capture component 110 is centered on a doorway at the opposite end of the hallway. As such, a distant object of interest (representing a person) is confined to a narrow ROI between and parallel to the floor and ceiling in the image 400. More specifically, as Figure 4 , the head of the person is confined to a horizontal region spanning 25% of the total height of the image 400, while the entire person is confined to a horizontal region spanning 33% of the total height of the image 400. Since the FOV 112 is centered between the floor and the ceiling, the ROI is also centered in the middle of the image 400. However, the location of the ROI relative to the image 400 may vary depending on the positioning of the FOV 112 of the image capture component 110.

[0060] Figure 5 1 is a block diagram of an example object detection network 500 according to some implementations. The object detection network 500 is configured to receive an input image 501 and generate a number of inferences 502-505 based on the input image 501. In some implementations, the object detection network 500 may be Figure 1 For example, refer to the image analysis unit 120. Figure 1 , input image 501 may be an example of image 102 , and inferences 502 - 505 may be examples of inferences 103 .

[0061] The object detection network 500 includes an FPN 510, a plurality of network heads 522-525, and a ROI extraction component 530. The FPN 510 is configured to extract a plurality of feature maps P2-P5 from an input image 501 based at least in part on a CNN. In some implementations, the FPN 510 may be Figure 3 Thus, FPN 510 may include a bottom-up path (such as bottom-up path 312) and a top-down path (such as top-down path 314). Figure 3 As described above, each of the feature maps P2-P5 is associated with a corresponding pyramid level of FPN 510. Figure 5 In the example of FIG. 5 , FPN 510 is shown as including 4 pyramid levels. In an actual implementation, FPN 510 may include more than Figure 5 Fewer or more pyramid levels than those depicted in .

[0062] exist Figure 5In the example of FIG. 5 , the ROI extraction component 530 is configured to select a region of the feature map P2 associated with the second pyramid level of the FPN 510 for distant object detection. For example, the selected region may coincide with the ROI in which the distant objects in the expected input image 501 are confined (such as the reference Figure 4 In some aspects, the ROI extraction component 530 may output only a subset of activations in the feature map P2 bounded by the selected region (ROI P2 ). For example, activate the subset ROI P2 can span the width of feature map P2. In some implementations, the subset ROI is activated P2 It can represent ~25% of all activations in feature map P2. In some other implementations, the activation subset ROI P2 It can represent ~33% of all activations in feature map P2.

[0063] The network head 522 is configured to activate the subset ROI of the feature map P2 P2 The object detection operation is performed, and the network heads 523-525 are configured to perform the object detection operation on the feature maps P3-P5, respectively. As such, each of the network heads 522-525 is also associated with a corresponding pyramid level of the FPN 510. In some aspects, each of the network heads 522-525 can be configured to infer a bounding box associated with an object of interest in the input image 501 (such as a reference Figure 3 ). In some implementations, each of the network heads 522-525 may include a classification subnetwork and a box regression subnetwork. For example, the classification subnetwork may infer or indicate the probability of detecting an object of interest at each box location, and the box regression subnetwork may infer or regress the offset of each box relative to a nearby object of interest (if detected).

[0064] As reference Figure 3 As described above, a network head associated with a lower pyramid level, such as network head 522, can detect objects based on smaller or finer features. As such, a distant object in input image 501 is more likely to be detected by network head 522 than by any of the remaining network heads 523-525. Since the object detection operation performed by network head 522 is limited to the activation subset ROI P2 (rather than the entire feature map P2), so the network head 522 can achieve Figure 3 The network head 322 of the object detection network 500 can be configured to have a greater number of neural network filters without causing the object detection network 500 to exceed the memory budget of the object detection network 300. As a result, given the same memory budget, the object detection network 500 is able to detect distant objects more accurately than the object detection network 300.

[0065] exist Figure 5 In the example of , the ROI extraction component 530 is associated only with the second pyramid level of FPN 510. In other words, only the size of feature map P2 is reduced for processing by network head 522. In some other implementations, the ROI extraction component 530 can be associated with one or more higher pyramid levels of FPN 510 instead of or in addition to the second pyramid level. For example, the ROI extraction component 530 can reduce the size of any of the remaining feature maps P3-P5 in the same or similar manner as feature map P2. In such an implementation, the number of neural network filters implemented by the remaining network heads 523-525 can also be increased without exceeding the memory budget of the object detection network 500.

[0066] Figure 6 An example ROI 610 is shown that can be extracted from the example feature map 600. Figure 6 In the example of 1,1 -A 6,6 In practical implementations, feature map 600 may have any suitable size. In some implementations, feature map 600 may be Figure 5 An example of a feature map P2, and ROI 610 may include an activation subset ROI P2 In some aspects, the ROI 610 of the feature map 600 may be similar to the ROI of the input image in which the distant object is confined (such as the ROI in which the entire human body is confined). Figure 4 The ROI of the input image 400) overlaps.

[0067] like Figure 6 As shown in FIG, the feature map 600 is subdivided into three horizontal segments (referred to as the upper segment, the middle segment, and the lower segment). More specifically, the upper segment includes the top two rows of activations A 1,1 -A 1,6 and A 2,1 -A 2,6 , the middle section includes the middle two rows of activation A 3,1 -A 3,6 and A 4,1 -A 4,6 , and the lower segment includes the bottom two rows of activation A 5,1 -A 5,6 and A 6,1 -A 6,6 .exist Figure 6 In the example of FIG. 6 , ROI 610 represents the middle segment, which includes 33% of the total activations in feature map 600 (shown in gray shading). However, ROI 610 may include more than Figure 6600. For example, in some other implementations, ROI 610 may include 25% of the total activations in feature map 600. Figure 5 , the object detection operation performed by the network head 522 can be limited to the ROI 610 of the feature map 600. For example, the ROI extraction component 530 can only activate the subset A 3,1 -A 3,6 and A 4,1 -A 4,6 The output is sent to the network head 522 for long-range object detection.

[0068] Figure 7 1 is a block diagram of another example object detection network according to some implementations. The object detection network 700 is configured to receive an input image 701 and generate a number of inferences 702-705 based on the input image 701. In some implementations, the object detection network 700 may be Figure 1 For example, refer to the image analysis unit 120. Figure 1 , input image 701 may be an example of image 102 , and inferences 702 - 705 may be examples of inferences 103 .

[0069] The object detection network includes FPN 710, several network heads 722-725, ROI extraction component 730, and distant object backbone 740. FPN 710 is configured to extract several feature maps P3-P5 from input image 701 based at least in part on CNN. In some implementations, FPN 710 can be Figure 3 Thus, FPN 710 may include a bottom-up path (such as bottom-up path 312) and a top-down path (such as top-down path 314). Figure 3 As described above, each of the feature maps P3-P5 is associated with a corresponding pyramid level of FPN 710. Figure 7 In the example of FIG. 7 , FPN 710 is shown as including 4 pyramid levels. However, unlike top-down path 314, the top-down path of FPN 710 does not produce a feature map associated with the second pyramid level (such as feature map P2 output by FPN 310). In actual implementations, FPN 710 may include more than Figure 7 Fewer or more pyramid levels than those depicted in .

[0070] The ROI extraction component 730 is configured to select a region of the input image 701 for long-distance object detection. For example, the selected region may be an ROI (such as a reference region) in which a long-distance object in the input image 701 is expected to be confined. Figure 4In some aspects, the ROI extraction component 730 may output only a subset of pixel values ​​in the input image 701 bounded by the selected region (ROI in ). For example, pixel value subset ROI in The pixel value subset ROI may span the width of the input image 701. In some implementations, the pixel value subset ROI in It can represent ~25% of all pixel values ​​in the input image 701. In some other implementations, the pixel value subset ROI in ∼33% of all pixel values ​​in the input image 701 can be represented.

[0071] The distant object backbone 740 is configured to subset the pixel values ​​of the ROI in Extract feature map P2 associated with the second pyramid level of FPN 710. In some implementations, distant object backbone 740 can be a feed-forward CNN with two network stages (similar to the first two network stages of the bottom-up path of FPN 710). Since the feature extraction operation performed by distant object backbone 740 is limited to a subset of pixel values ​​ROI in (rather than the entire input image 701), so the feature map P2 generated by the distant object backbone 740 may include more Figure 3 The feature map P2 produced by the FPN 310 of the distant object backbone 740 has fewer activations. As such, the distant object backbone 740 is able to implement a greater number of neural network filters than the first two network stages of the bottom-up path 312 of the FPN 310 without causing the object detection network 700 to exceed the memory budget of the object detection network 300.

[0072] The network heads 722-725 are configured to perform object detection operations on the feature maps P2-P5, respectively. As such, each of the network heads 722-725 is also associated with a corresponding pyramid level of the FPN 710. In some aspects, each of the network heads 722-725 can be configured to infer a bounding box associated with an object of interest in the input image 701 (such as a reference Figure 3 ). In some implementations, each of the network heads 722-725 may include a classification subnetwork and a box regression subnetwork. For example, the classification subnetwork may infer or indicate the probability of detecting an object of interest at each box location, and the box regression subnetwork may infer or regress the offset of each box relative to a nearby object of interest (if detected).

[0073] As reference Figure 3As described above, a network head associated with a lower pyramid level, such as network head 722, can detect objects based on smaller or finer features. As such, a distant object in input image 701 is more likely to be detected by network head 722 than by any of the remaining network heads 723-725. Figure 3 The feature map P2 produced by FPN 310 is smaller (or has fewer activations), so network head 722 is able to implement the same (or more) number of neural network filters as network head 322 without causing object detection network 700 to exceed the memory budget of object detection network 300. As a result, given the same memory budget, object detection network 700 is able to detect distant objects more accurately than object detection network 300.

[0074] exist Figure 7 In the example of FIG. 7 , the distant object backbone 740 is associated only with the second pyramid level of the FPN 710. In other words, only the size of the feature map P2 is reduced for processing by the network head 722. In some other implementations, the distant object backbone 740 can be associated with one or more higher pyramid levels of the FPN 710 instead of or in addition to the second pyramid level. For example, the distant object backbone 740 can be used to select a pixel value subset ROI based on the same or similar manner as the feature map P2. in Any of the remaining feature maps P3-P5 is generated. In such an implementation, the number of neural network filters implemented by the remaining network heads 523-525 can also be increased without exceeding the memory budget of the object detection network 500.

[0075] Aspects of the present disclosure further recognize that in Figure 7 In the example of , the higher pyramid levels of FPN 710 can only be used to detect objects of interest that are larger or closer to the image capture device (compared to the objects detected by network head 722). As such, the number of neural network filters implemented by FPN 710 (compared to the number of neural network filters implemented by FPN 310) can be reduced without sacrificing accuracy in detecting larger objects. In some implementations, the bottom-up path of FPN 710 can implement fewer neural network filters than the bottom-up path 312 of FPN 310. In such an implementation, the distant object backbone 740 can implement even more filters than the first two network levels of the bottom-up path 312 without causing the object detection network 700 to exceed the memory budget of the object detection network 300. Such an implementation can result in even greater accuracy in distant object detection by the object detection network 700.

[0076] Figure 8An example ROI 810 is shown that can be extracted from an example input image 800. Figure 8 In the example of FIG. 8 , the input image 800 is depicted as pixel values ​​P 1,1 -P 8,8 In practical implementations, the input image 800 may have any suitable size. In some implementations, the input image 800 may be Figure 7 An example of an input image 701, and the ROI 810 may include a pixel value subset ROI in In some aspects, the ROI 810 of the input image 800 may be an ROI in which a distant object is confined (such as a person's head is confined). Figure 4 ROI of the input image 400).

[0077] like Figure 8 As shown in , the input image 800 is subdivided into three horizontal segments (referred to as the upper segment, the middle segment, and the lower segment). More specifically, the upper segment includes the top three rows of pixel values ​​P 1,1 -P 1,8 , P 2,1 -P 2,8 and P 3,1 -P 3,8 , the middle section includes the middle two rows of pixel values ​​P 4,1 -P 4,8 and P 5,1 -P 5,8 , and the lower segment includes the bottom three rows of pixel values ​​P 6,1 -P 6,8 , P 7,1 -P 7,8 and P 8,1 -P 8,8 .exist Figure 8 In the example of FIG. 8 , ROI 810 represents the middle segment, which includes 25% of the total activations in input image 800 (shaded in gray). However, ROI 810 may include more than Figure 8 800. For example, in some implementations, ROI 810 may include 33% of the total activations in input image 800. For example, referring to Figure 7 , the feature extraction operation performed by the distant object backbone 740 may be limited to the ROI 810 of the input image 800. For example, the ROI extraction component 730 may only extract the pixel value subset P 4,1 -P 4,8 and P 5,1 -P 5,8 Output to object detection model 740 for feature extraction.

[0078] Fig. 9A block diagram of an example object detection system 900 is shown according to some implementations. In some implementations, the object detection system 900 may be Figure 1 The image analysis component 120 or Figure 5 900 is an example of an object detection network 500. More specifically, the object detection system 900 can be configured to infer the presence or location of an object of interest in an image (or video).

[0079] The object detection system 900 includes an image source interface 910, a processing system 920, and a memory 930. The image source interface 910 is configured to receive an image from an image source such as Figure 1 The memory 930 may include: an image data buffer 931 for storing the received input image; and a feature map buffer 932 for storing one or more feature maps generated by the object detection system 900 as a result of generating an inference.

[0080] The memory 930 may also include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, hard drive, etc.) that may store at least the following software (SW) modules: a feature map extraction SW module 933 for extracting a plurality of feature maps from an input image based on an FPN having a plurality of pyramid levels, wherein each feature map of the plurality of feature maps is associated with a corresponding pyramid level of the plurality of pyramid levels; ROI selection SW module 934, which is used to select a region of a first feature map among a plurality of feature maps for long-distance object detection; and An object detection SW module 935 for performing an object detection operation on each of the plurality of feature maps based on a corresponding network head, such that the object detection operation performed on the first feature map is limited to a selected region. Each software module includes instructions that, when executed by processing system 920 , cause object detection system 900 to perform a corresponding function.

[0081] The processing system 920 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the object detection system 900 (such as in the memory 930). For example, the processing system 920 may execute a feature map extraction SW module 933 to extract multiple feature maps from an input image based on an FPN having multiple pyramid levels, wherein each feature map in the multiple feature maps is associated with a corresponding pyramid level in the multiple pyramid levels. The processing system 920 may also execute a ROI selection SW module 934 to select an area of ​​a first feature map in the multiple feature maps for long-distance object detection. The processing system 920 may further execute an object detection SW module 935 to perform an object detection operation on each feature map in the multiple feature maps based on the corresponding network head, so that the object detection operation performed on the first feature map is limited to the selected area.

[0082] Fig.10 1 is a block diagram showing another example object detection system 1000 according to some implementations. In some implementations, the object detection system 1000 may be Figure 1 The image analysis component 120 or Figure 7 1000 is an example of an object detection network 700. More specifically, object detection system 1000 can be configured to infer the presence or location of an object of interest in an image (or video).

[0083] The object detection system 1000 includes an image source interface 1010, a processing system 1020, and a memory 1030. The image source interface 1010 is configured to receive an image from an image source such as Figure 1 The memory 1030 may include: an image data buffer 1031 for storing the received input image; and a feature map buffer 1032 for storing one or more feature maps generated by the object detection system 1000 as a result of generating an inference.

[0084] The memory 1030 may also include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, hard drive, etc.) that may store at least the following software (SW) modules: ROI selection SW module 1033, which is used to select the region of the input image for long-range object detection; ROI feature map extraction SW module 1034, which is used to extract a first feature map of multiple feature maps from a selected region of the input image based on the backbone CNN; An image feature map extraction SW module 1035 for extracting one or more second feature maps of a plurality of feature maps from an input image based on an FPN having a plurality of pyramid levels, wherein each feature map of the plurality of feature maps is associated with a corresponding pyramid level of the plurality of pyramid levels; and An object detection SW module 1036 is configured to perform an object detection operation on each of the plurality of feature maps based on a corresponding network head. Each software module includes instructions that, when executed by processing system 1020 , cause object detection system 1000 to perform a corresponding function.

[0085] The processing system 1020 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the object detection system 1000 (such as in the memory 1030). For example, the processing system 1020 may execute the ROI selection SW module 1033 to select an area of ​​the input image for long-distance object detection. The processing system 1020 may further execute the ROI feature map extraction SW module 1034 to extract a first feature map of a plurality of feature maps from the selected area of ​​the input image based on the backbone CNN. The processing system 1020 may execute the image feature map extraction SW module 1035 to extract one or more second feature maps of a plurality of feature maps from the input image based on an FPN having a plurality of pyramid levels, wherein each feature map of the plurality of feature maps is associated with a corresponding pyramid level of the plurality of pyramid levels. The processing system 1020 may further execute the object detection SW module 1036 to perform an object detection operation on each feature map of the plurality of feature maps based on a corresponding network head.

[0086] Fig.11 An illustrative flow chart depicting example object detection operations 1100 according to some implementations is shown. In some implementations, the example operations 1100 can be performed by an object detection system such as Figure 1 The image analysis component 120 or Figure 5 An object detection network 500 of the present invention is implemented to infer the presence or location of an object of interest in an image (or video).

[0087] An object detection system receives an input image (1110). The object detection system extracts a plurality of feature maps from the input image based on an FPN having a plurality of pyramid levels, wherein each feature map in the plurality of feature maps is associated with a corresponding pyramid level in the plurality of pyramid levels (1120). The object detection system selects an area of ​​a first feature map in the plurality of feature maps for long-range object detection (1130). The object detection system performs an object detection operation on each feature map in the plurality of feature maps based on a corresponding network head, such that the object detection operation performed on the first feature map is limited to the selected area (1140).

[0088] In some implementations, the FPN may include a bottom-up path that generates a plurality of feature maps in order of increasing semantic value based on an input image. In some other implementations, the FPN may include: a bottom-up path that generates a plurality of intermediate feature maps in order of increasing semantic value based on an input image; and a top-down path that generates a plurality of feature maps in order of increasing spatial resolution based on the plurality of intermediate feature maps.

[0089] In some implementations, the first feature map may be associated with a lowest pyramid level of the plurality of pyramid levels. In some aspects, the first feature map may be horizontally subdivided into three non-overlapping segments, and the selected region of the first feature map may represent a middle segment of the three segments. In some implementations, the selected region may intersect with a center of the first feature map. In some implementations, the selected region may include 25% or less of the first feature map.

[0090] In some implementations, each of the network heads may include a classification subnetwork and a box regression subnetwork, wherein the classification subnetwork indicates a probability of detecting an object in each of a plurality of boxes mapped to a corresponding feature map, and the box regression subnetwork regresses a corresponding offset of each of the plurality of boxes relative to the object.

[0091] Fig.12 An illustrative flow chart depicting another example object detection operation according to some implementations is shown. In some implementations, the example operation 1200 can be performed by an object detection system such as Figure 1 The image analysis component 120 or Figure 7 An object detection network 700 of the present invention is implemented to infer the presence or location of an object of interest in an image (or video).

[0092] The object detection system receives an input image (1210). The object detection system selects an area of ​​the input image for long-distance object detection (1220). The object detection system extracts a first feature map of a plurality of feature maps from the selected area of ​​the input image based on a backbone CNN (1230). The object detection system also extracts one or more second feature maps of the plurality of feature maps from the input image based on an FPN having a plurality of pyramid levels, wherein each feature map of the plurality of feature maps is associated with a corresponding pyramid level of the plurality of pyramid levels (1240). The object detection network further performs an object detection operation on each feature map of the plurality of feature maps based on a corresponding network head (1250).

[0093] In some implementations, the FPN may include a bottom-up path that generates multiple feature maps in order of increasing semantic value based on an input image. In some other implementations, the FPN may include: a bottom-up path that generates multiple intermediate feature maps in order of increasing semantic value based on an input image; and a top-down path that generates multiple feature maps in order of increasing spatial resolution based on multiple intermediate feature maps. In some implementations, each level of the backbone CNN may have a greater number of neural network filters than the corresponding level of the bottom-up path associated with the same pyramid level of the FPN.

[0094] In some implementations, the first feature map may be associated with a lowest pyramid level of the plurality of pyramid levels. In some aspects, the input image may be horizontally subdivided into three non-overlapping segments, and the selected region of the input image may represent the middle of the three segments. In some implementations, the selected region may intersect the center of the input image. In some implementations, the selected region may include 25% or less of the input image.

[0095] In some implementations, each of the network heads may include a classification subnetwork and a box regression subnetwork, wherein the classification subnetwork indicates a probability of detecting an object in each of a plurality of boxes mapped to a corresponding feature map, and the box regression subnetwork regresses a corresponding offset of each of the plurality of boxes relative to the object.

[0096] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0097] Moreover, it will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician can implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as departing from the scope of the present disclosure.

[0098] The methods, sequences or algorithms described in conjunction with the aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or in any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated with the processor.

[0099] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the present disclosure as set forth in the appended claims. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for object detection, the method comprising: receiving an input image; extracting a plurality of feature maps from the input image based on a feature pyramid network (FPN) having a plurality of pyramid levels, each feature map in the plurality of feature maps being associated with a corresponding pyramid level in the plurality of pyramid levels; selecting a region of a first feature map among the plurality of feature maps for long-range object detection; as well as An object detection operation is performed on each of the plurality of feature maps based on a corresponding network head, such that the object detection operation performed on the first feature map is limited to the selected region.

2. The method of claim 1, wherein the FPN comprises a bottom-up path that generates the plurality of feature maps in order of increasing semantic value based on the input image.

3. The method according to claim 1, wherein the FPN comprises a bottom-up path and a top-down path, wherein the bottom-up path generates a plurality of intermediate feature maps in an order of increasing semantic value based on the input image, and the top-down path generates the plurality of feature maps in an order of increasing spatial resolution based on the plurality of intermediate feature maps. 4 . The method of claim 1 , wherein the first feature map is associated with a lowest pyramid level of the plurality of pyramid levels. 5 . The method of claim 1 , wherein the first feature map is horizontally subdivided into three non-overlapping segments, and the selected region of the first feature map represents a middle segment of the three segments. The method according to claim 5 , wherein the selected region intersects with a center of the first feature map.

7. The method of claim 5, wherein the selected region comprises 25% or less of the first feature map.

8. The method of claim 1 , wherein each of the network heads comprises a classification subnetwork and a box regression subnetwork, the classification subnetwork indicating a probability of detecting an object in each of a plurality of boxes mapped to the corresponding feature map, and the box regression subnetwork regressing a corresponding offset of each of the plurality of boxes relative to the object.

9. A method for object detection, the method comprising: receiving an input image; selecting a region of the input image for long-range object detection; Extracting a first feature map from a selected region of the input image based on a backbone convolutional neural network (CNN); extracting one or more second feature maps of the plurality of feature maps from the input image based on a feature pyramid network (FPN) having a plurality of pyramid levels, each feature map of the plurality of feature maps being associated with a corresponding pyramid level of the plurality of pyramid levels; as well as An object detection operation is performed on each of the plurality of feature maps based on a corresponding network head.

10. The method of claim 9, wherein the FPN comprises a bottom-up path that generates the plurality of feature maps in order of increasing semantic value based on the input image.

11. The method according to claim 9, wherein the FPN comprises a bottom-up path and a top-down path, wherein the bottom-up path generates a plurality of intermediate feature maps based on the input image in order of increasing semantic value, and the top-down path generates the one or more second feature maps based on the plurality of intermediate feature maps in order of increasing spatial resolution.

12. The method of claim 11, wherein each level of the backbone CNN has a greater number of neural network filters than a corresponding level of the bottom-up path associated with the same pyramid level of the FPN.

13. The method of claim 9, wherein the first feature map is associated with a lowest pyramid level of the plurality of pyramid levels.

14. The method of claim 9, wherein the input image is subdivided horizontally into three non-overlapping segments, and the selected region of the input image represents a middle segment of the three segments. The method of claim 14 , wherein the selected region intersects a center of the input image.

16. The method of claim 14, wherein the selected area comprises 25% or less of the input image.

17. The method of claim 9, wherein each of the network heads comprises a classification subnetwork and a box regression subnetwork, the classification subnetwork indicating a probability of detecting an object in each of a plurality of boxes mapped to the corresponding feature map, and the box regression subnetwork regressing a corresponding offset of each of the plurality of boxes relative to the object.

18. An object detection system, the object detection system comprising: Processing systems; as well as a memory storing instructions that, when executed by the processing system, cause the object detection system to: receiving an input image; selecting a region of the input image for long-range object detection; Extracting a first feature map from a selected region of the input image based on a backbone convolutional neural network (CNN); extracting one or more second feature maps of the plurality of feature maps from the input image based on a feature pyramid network (FPN) having a plurality of pyramid levels, each feature map of the plurality of feature maps being associated with a corresponding pyramid level of the plurality of pyramid levels; as well as An object detection operation is performed on each of the plurality of feature maps based on a corresponding network head.

19. The object detection system of claim 18, wherein the first feature map is associated with a lowest pyramid level of the plurality of pyramid levels.

20. The object detection system of claim 19, wherein each level of the backbone CNN has a greater number of neural network filters than a corresponding level of the bottom-up path associated with the same pyramid level of the FPN.

Images