Neural network and classifier selection system and method

Abstract

A high-resolution image object classification system and method includes: a proposal component configured to receive a first data set associated with a scene, the first data set including at least one image of the scene; a multi-level neural network including a plurality of neural networks, each neural network trained to receive a region of interest and output an object classification based on an associated resource allocation; and an attention coordinator configured to determine regions of interest in the image based on available system resources and assign each determined region to one of the plurality of neural networks in the multi-scale neural network. The system can be configured to optimize the probability of detecting an object in the image while minimizing the number of pixels processed by the multi-scale neural network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 850,491, filed on May 20, 2019, and entitled “Neural Network and Classifier Selection System and Method,” which is hereby incorporated by reference in its entirety. Technical Field

[0003] One or more embodiments of the present disclosure relate generally to imaging systems, and more particularly, for example, to machine learning systems and methods for image classification systems. Background Art

[0004] In the field of image processing, there has been a need for efficient and reliable methods to detect and classify objects of interest within the field of view (e.g., scene) of an imaging device. Traditional "smart cameras" combine machine vision imaging components with a single-board computer running rule-based image processing software. These systems are used for simple problems such as barcode reading or identifying specific features of known objects.

[0005] Machine learning systems have been implemented to provide more sophisticated image analysis. In one approach, various images of an object of interest are collected into a training data set for training a neural network to classify the object. The training images may be generated by capturing images of the object with a camera at various angles and in various settings. The training data set typically contains thousands of images for each object classification, and can be time consuming, expensive, and cumbersome to generate and update. The trained neural network may be loaded onto a server system that receives and classifies images from imaging devices on the network. In some embodiments, the trained neural network may be loaded onto an imaging system.

[0006] Simplified machine vision and image classification systems are available, but such systems are unable to run robust trained neural networks and are difficult to adapt to a variety of user scenarios. In actual implementations, limitations on memory, processing, and other system resources often cause system designers to produce classification systems that are targeted to specific tasks. For example, neural networks are typically trained for specific classification tasks and implemented to allow real-time operation within system constraints. In view of the foregoing, there is a continuing need for improved object detection and classification solutions that are easily adaptable to new use cases and provide performance or other advantages over conventional systems. Summary of the invention

[0007] Various systems and methods are provided to implement training neural networks with imaging systems. A novel neural network object classification system enables users to rapidly accelerate learning techniques, frameworks, and tools.

[0008] A high-resolution image object classification system and method may include a proposal component configured to receive a first data set associated with a scene, the first data set including at least one image of the scene; a multi-level neural network including a plurality of neural networks, each neural network trained to receive a region of interest and output an object classification based on an associated resource allocation; and an attention coordinator configured to determine regions of interest in the image based on available system resources and assign each determined region to one of the plurality of neural networks of the multi-scale neural network. The system may be configured to optimize the probability of detecting an object in the image while minimizing the number of pixels processed by the multi-scale neural network.

[0009] In some embodiments, the systems and methods include an end-to-end system for detecting objects using neural network object detection and classification. A video feed representing an image of a scene is provided to the system from one or more input devices. An external system (e.g., a radar system) may also be used to provide information about proposed locations of objects of interest. Optionally, an internal motion detector may be used to provide proposed locations of objects of interest. A random search algorithm may be used with a large area neural network to find proposed objects of interest (e.g., when no other proposed locations have been identified). A convolutional neural network attention coordinator may optimize the available neural networks to process the proposals using multiscale neural network processing. Targets are identified from multiscale neural network processing and / or motion detection algorithms. Short-term target memory is applied to predict new target locations, eliminate temporary detection adverse consequences, and track through occlusion. The system may output a video or image with a box around the detected object of interest.

[0010] The scope of the present invention is defined by the claims, which are incorporated herein by reference. By considering the following detailed description of one or more embodiments, those skilled in the art will more fully understand the embodiments of the present invention, and realize the additional advantages thereof. Reference will be made to the accompanying drawings, which will first be briefly described. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example neural network target classifier is shown in accordance with one or more embodiments of the present disclosure.

[0012] Figure 2 An example implementation of a neural network target classifier according to one or more embodiments of the present disclosure is shown.

[0013] Figure 3 An example input device for use with a neural network object classifier is shown in accordance with one or more embodiments of the present disclosure.

[0014] Figure 4A An example neural network training process according to one or more embodiments of the present disclosure is shown.

[0015] Figure 4B Example trained neural network operations are shown in accordance with one or more embodiments of the present disclosure.

[0016] Figure 5 An example implementation of a neural network target classifier according to one or more embodiments of the present disclosure is shown.

[0017] Figure 6 Example operations of a neural network target classifier according to one or more embodiments of the present disclosure are shown.

[0018] Embodiments of the present disclosure and their advantages may be best understood by referring to the following detailed description.It should be appreciated that like reference numerals are used to identify like elements shown in one or more of the Figures. DETAILED DESCRIPTION

[0019] Aspects of the present disclosure relate generally to image classification, and more specifically, for example, to object detection and classification. In various embodiments, an improved imaging system receives video of a scene and / or other data from the scene related to object locations. For example, an imaging device includes an image sensor, a visual processing unit, and an interface for capturing video for further processing by a neural network object classifier. In some embodiments, the captured images include high-resolution images and video data. The neural network object classifier receives the data and generates proposals for potential object locations, analyzes the proposals based on available resources through an attention coordinator, identifies regions of interest for further processing, and provides each region of interest to an appropriate neural network for object detection and classification.

[0020] Processing high resolution images by neural networks in conventional image classification systems typically involves highly complex neural networks that require processing resources that may exceed the resources of the system and / or require processing with reduced accuracy and detail. The systems and methods of the present disclosure overcome these limitations, which enables processing of large resolution images or videos of objects of interest at a high sensitivity level. The present disclosure includes methods for processing large image and video files by reducing the received information to a manageable computational area. In some embodiments, an optimized neural network generates proposed regions of interest for classification and also receives proposal requests from an external input device. The method can also select an optimal neural working classifier to process the proposal and classify the object of interest in the proposed region.

[0021] In various embodiments, an image classification system with improved processing is disclosed that is more efficient and effective than conventional systems in achieving system goals. The image classification system can receive one or more high-resolution video streams and efficiently process the high-resolution video streams in real time (or in some embodiments offline) with a high level of detail and accuracy within the constraints of available system resources. The system is also configured to simultaneously process other data streams associated with the captured scene, such as radar data or audio data, and output annotated video (e.g., bounding boxes on video images with associated object classification labels) that identify and classify objects that appear.

[0022] refer to Figure 1 , an example object detection and classification system 100 will be described according to one or more embodiments of the present disclosure. System 100 includes one or more input devices 110 configured to sense characteristics of a scene 112 and generate corresponding data. Input devices 110 may include visible spectrum imaging devices configured to capture images and / or video of scene 112, radar systems configured to generate radar data signals representative of physical objects in scene 112, infrared imaging systems configured to detect infrared radiation from scene 112, and audio input systems configured to sense sounds produced by scene 112 and / or other input devices.

[0023] The data captured by the input device 110 is provided to the neural network target classifier 120 for object detection and classification. In some embodiments, the captured data 114 includes one or more high-resolution data streams. The captured data 114 may include, for example, a video feed from one or more cameras in a surveillance system. The captured data 114 may also include data identifying the location of an object of interest, such as data from a motion capture and / or object detection component of the input device 110, data from a radar system that identifies the location of a detected object, and / or other object location data. The proposal unit 122 may also include object detection, motion detection components, and other components configured to detect the presence and location of an object in the captured data. For example, a random search algorithm may be used with a neural network to find a proposed object of interest in a frame of the captured data 114.

[0024] Many systems have memory, processing, bandwidth, and real-time constraints that make analyzing each proposed object inefficient and / or impractical. The neural network target classifier system 120 includes an attention coordinator 124 that is configured to optimize a neural network to process proposals generated by the proposal unit 122. In some embodiments, the attention coordinator 124 is a neural network that allocates available system resources to maximize object detection and classification results. The attention coordinator 124 generates an attention region of interest (ROI) 126 that includes new target location boundaries and allocates the appropriate neural network in the multi-scale neural network 128 to process each target location boundary. The multi-scale neural network 128 includes neural networks of different sizes and complexities, allowing processing to scale to the input data. The neural network target classifier 120 outputs target object detection and classification results, which may include a video and / or image output 130 with boxes (e.g., boxes 132, boxes 134, and boxes 136) around the detected objects of interest. For example, the output may include a target identifier, box location and size, target classification (e.g., drone, person, not_classified), and other information.

[0025] The neural network object classifier 120 is configured to process high-resolution images for object detection and classification using a neural network trained to efficiently process smaller images. The attention coordinator 124 is configured to generate an attention RIO 126 that defines a portion of the high-resolution image to be processed at each time step (e.g., a frame or sequence of frames of data). In some embodiments, the attention coordinator 124 is configured to identify tracking areas associated with known objects previously detected in previous images. The attention coordinator 124 can also be configured to identify areas for new object discovery. The proposal unit 122 analyzes available input data from the input device 110 and generates proposals for areas to be analyzed for new objects. The proposal can be received from one or more input devices 110 via a local interface (e.g., an image capture device with object detection capabilities) or determined by one or more modules of the proposal unit 122. For example, the proposal unit 122 can detect motion in an image sequence and propose to search for areas to identify new objects.

[0026] Now refer to Figure 2Additional details of the object detection and classification system are described. The object detection and classification system 200 includes multiple input devices 210, which may include a signal processing device 212 (e.g., a radar system), a video capture device 214 (e.g., a camera in a house surveillance system or a camera mounted on a drone for capturing aerial video), and an image capture device 216 (e.g., a visible light or infrared camera). Each input device 210 is configured to capture data of a scene and provide corresponding data to a neural network object classifier 240 via a communication network and / or other communication connection. The input devices 210 may also include processing components for identifying and / or tracking objects of interest. For example, a radar system may provide data representing the location of an object or structure within a scene. The video processing device may include motion detection or other object detection components.

[0027] Now refer to Figure 3 An example implementation of an input device is described in more detail. The input device 300 includes a device configured to capture data related to a scene 310 and to send data to the scene 310. Figure 2 The input device 300 may include any device capable of sensing characteristics of one or more objects in the scene 310, such as an image processing and / or signal processing unit. In the illustrated embodiment, the input device 300 includes a radar system 320, a video capture device 330, and a video capture device 340. The radar system 320 is configured to transmit a series of pulses or continuous waves into the scene 310 and detect pulses / waves reflected back to the radar system 320 from objects in the scene 310. The radar system 320 generates signal data 322 representing objects in the scene, and the radar processing 324 is configured to analyze the signal data 322 to identify the location of the objects within the scene 310. The object locations are output to the proposal unit 230 (see Figure 2 ) for further processing. Video capture device 330 may include an imaging sensor configured to capture visible or infrared images of scene 310 and generate a stream of video images 332. Video images 332 are analyzed by video processing unit 334, which provides motion detection and / or object detection for the captured video. The identified objects are provided to proposal unit 230 (see Figure 2 ). Input device 300 may also include a video capture device 340 including an imaging sensor configured to capture a visible light or infrared image 342 of scene 310 and generate a video image 342 that is transmitted to proposal unit 230 (see Figure 2 ) of the corresponding video stream 344. The video stream 344 can be analyzed by the proposal unit 230 to detect motion and objects.

[0028] Return to reference Figure 2In various embodiments, the input device 210 outputs a data stream to a proposal unit 230 of a neural network object classifier 240. The proposal unit 230 is configured to receive data from the input device 210 and output identifications of proposed target objects of interest (e.g., regions of a captured image that may include the target object of interest). The proposal unit 230 may include components for receiving proposed target object locations from the input device 210 and / or components for analyzing the input data to identify additional proposed target object locations. In the illustrated embodiment, the proposal unit includes a proposal unit application programmable interface (API) 232 for facilitating communication between the input device 210 and the neural network object classifier 240 (e.g., forwarding externally generated proposals to the attention coordinator 250). The proposal unit 230 may also include an image processing unit 234 for analyzing captured image data for objects of interest, a motion target indicator 236 for detecting and / or tracking motion of one or more objects in the input stream (e.g., to generate motion-based proposals), a random search component 238 configured to search the captured images for potential objects and generate random search proposals, and a short-term target memory 239 for tracking areas previously identified in the input data stream that may include detected objects (e.g., tracked objects that may not be in the field of view, doorways of buildings, high-traffic areas, or other areas where objects were previously detected). The short-term target memory 239 may generate predictive proposals for the attention coordinator 250.

[0029] The target object proposals are provided to the attention coordinator 250, which is configured to optimize the region of interest for further processing through a region optimization process 252 and a process for tracking / assigning computational constraints 254. The target object proposals may include a bounding box for a potential target, such as a rectangle defined in a portion of an image, defining an area that may contain an object. In one embodiment, the attention coordinator 250 is configured to optimize the region of interest for small target acquisition and high frame rate target following. The attention coordinator 250 identifies an attention region of interest (ROI) 260, which includes a set of modified regions (e.g., boxes 262, boxes 264, and boxes 266) for processing by the multi-scale neural network 270. In some embodiments, the attention coordinator 250 is configured to maximize the probability of object detection (e.g., the probability of detecting all objects in the image) while minimizing the total computational processing (e.g., the number of input pixels processed by the multi-scale neural network 270). The attention ROI can be optimized to cover all proposals, for example, by clustering the proposal regions and allocating available multi-scale neural network resources to optimize processing.

[0030] The multi-scale neural network 270 includes multiple neural networks optimized to process regions of different sizes. The attention ROI 260 assigns each attention region of interest (e.g., boxes 262, 264, and 266) to one of the neural networks in the multi-scale neural network 270. In the illustrated embodiment, the various neural networks are trained to process image regions of different sizes, including image sizes of 64×64, 128×128, 256×256, and 512×512, respectively. It should be understood that Figure 2 The input image sizes identified in are merely illustrative, and other neural network configurations and image sizes may be used in accordance with the teachings of the present disclosure. In various embodiments, the multiscale neural network includes neural networks of varying complexity, and the attention ROI 260 is selected to optimize target object detection and classification results using available computing resources. In some embodiments, the attention coordinator 250 implements a clustering algorithm that attempts to maximize the probability of detecting each target while minimizing the total number of pixels processed by the multiscale neural network 270.

[0031] In various embodiments, smaller image sizes can be processed more efficiently with fewer processing resources than larger images and can be assigned to neural networks configured to process images of similar sizes. In some embodiments, the attention coordinator 250 assigns small neural networks to process small target areas and assigns large neural networks to process large target areas. The attention coordinator 250 can also be configured to cluster proposed areas for efficient processing, including clustering one or more small target areas with large target areas to maximize the efficiency of the neural network. The attention ROI can also include large areas assigned to neural networks with smaller input image sizes. For example, the attention coordinator 250 may determine that a particular image area can be processed with less detail (e.g., large objects can be detected and classified without detailed analysis). The size of the particular image area can then be adjusted for input to the assigned neural network.

[0032] As previously described, various components in the illustrated embodiments may be implemented using trained neural networks. For example, the attention coordinator 250 may be implemented as a neural network trained to receive multiple region proposals and system resources and generate optimized regions of interest for further processing. The multi-scale neural network 270 may include multiple neural networks, each trained to detect and / or classify objects within a region of interest.

[0033] In various embodiments, the output includes an annotated video stream that identifies and labels the detected objects (e.g., via a labeled bounding box image overlay). In some embodiments, a portion of the output data is fed back to the system for processing the next time step. For example, a region defining an active object (e.g., a bounding box surrounding a detected object) can be provided to the attention coordinator 250 so that the same optimization can be performed at the next time step and / or the region can be combined with other proposed regions when defining a new attention ROI. Detected object regions can also be provided to the short-term target memory 239 to help identify potential regions for new objects. Regions without detected objects can also be provided to the attention coordinator 250, allowing resources to be directed to other proposed regions.

[0034] refer to Figure 4A and 4B , an example neural network will now be described. Figure 4A A training process for training a neural network 400 is shown. The neural network 400 includes at least one input layer 402, at least one hidden layer 404, and at least one output layer 406. The neural network 400 receives a training data set 410 including a series of labeled images representing objects in a region of interest and outputs an object classification for each image. The training data set 410 may include images captured from infrared, visible light, or other types of devices corresponding to input devices and / or data input to an object detection and classification system. In some embodiments, the training data set 410 includes one or more synthetically generated or modified images. The training data set 410 may also include other image classification input data available to the system (e.g., the output of another trained neural network). For example, the training data set 410 may include an attention region of interest generated by an attention coordinator.

[0035] For object classification, the image may include a region of interest from a captured image that includes an object to be identified. In one embodiment, training begins with a forward pass through the neural network 400, which may include feature extraction through multiple convolutional layers and pooling layers, followed by multiple fully connected hidden layers and image classification in the output layer 406. Next, a backward pass through the neural network 400 may be used to update the weighting parameters of the nodes of the neural network 400 to adjust for errors (e.g., misclassified objects) generated in the forward pass. In various embodiments, other types of neural networks and other training processes may be used in accordance with the present disclosure.

[0036] refer to Figure 4B, the trained neural network 450 can then be implemented in a runtime environment to classify objects in a region of interest 452 of an image. The runtime environment can include one or more embodiments of the systems and methods disclosed herein.

[0037] Now refer to Figure 5 An example embodiment of a neural network target classifier 500 is described. The neural network target classifier 500 can be implemented on one or more systems or servers, such as application servers that perform data processing and / or other software execution operations to generate, store, classify, and retrieve images. In some embodiments, the components of the neural network target classifier 500 can be distributed over a communication network such as a communication network 522. The communication network 522 can include one or more local area networks, such as a wireless local area network (WLAN), a wide area network (e.g., the Internet), and other wired or wireless communication paths suitable for facilitating communication between the components described herein. The neural network target classifier 500 includes a communication component 516 that is configured to facilitate communication with one or more input devices 520 via the communication network 522.

[0038] In various embodiments, the neural network target classifier 500 may operate as a general purpose image classification system, such as a cloud-based image classification system, or may be configured to operate in a dedicated system, such as a video surveillance system that stores video and images captured in real time from multiple image capture devices and identifies and classifies objects. The neural network target classifier 500 may be configured to receive one or more images (e.g., images captured from an infrared camera or visible light image of a video surveillance system) and other associated data (e.g., radar or audio data from one or more input devices 520) from one or more input devices 520 and process the associated object recognition / classification request.

[0039] In the illustrated embodiment, the neural network object classifier 500 includes one or more processors 504 that perform data processing and / or other software-implemented operations, and a database 502 that stores data used by the system. The processor 504 may include a logic device, a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other device that can be used by the neural network object classifier 500 to execute appropriate instructions (e.g., software instructions stored in the memory 506), including the neural network training component 508, the proposal unit component 510, the attention coordinator and attention ROI component 512, the trained multi-scale neural network component 514, and / or other applications. The memory 506 can be implemented in one or more memory devices (e.g., memory components) that store executable instructions, data, and information (including image data, video data, audio data, network information). In various embodiments, the neural network target classifier 500 can be configured to connect to various network devices, such as a desktop computer or network server, a mobile computing device (e.g., a mobile phone, a tablet computer, a laptop computer), or other computing devices having communication circuits (e.g., wireless communication circuits or wired communication circuits) for connecting to other devices in the neural network target classifier 500.

[0040] Communication component 516 may include circuits for communicating with other devices using various communication protocols. In various embodiments, for wired communication purposes, communication component 516 may be configured to communicate via a wired communication link (e.g., via a network router, switch, hub, or other network device). For example, a wired link may be implemented with a power line cable, a coaxial cable, an optical fiber cable, or other suitable cables or lines that support corresponding wired network technology. Communication component 516 may also be configured to be connected to a wired network and / or device interface via a wired communication component such as an Ethernet interface, a power line modem, and / or other suitable components for wired communication. Communication component 516 may also support proprietary wired communication protocols and interfaces.

[0041] Now refer to Figure 6 Describes example operations of an object detection and classification system using a neural network object classifier of the present disclosure. Process 600 may be performed on a networked neural network object classification system (e.g., Figure 5 The method is implemented in a dedicated system (e.g., a house video surveillance system) including one or more image capture devices, or in other image processing systems. In step 602, an input device captures data associated with a scene, including at least one image of the scene. The data may also include non-image data, such as radar data, audio data, and other sensory data.

[0042] In step 604, the captured data is analyzed to identify potential object regions based on image analysis, motion detection, radar signals, audio signals, and / or other data analysis. In step 606, these proposed object regions are then analyzed along with system resource information (memory, bandwidth, processing power) to derive attention regions of interest for further processing. An attention region of interest may include an area with two or more proposed object regions. For example, an attention region of interest may reduce the number of search regions by defining a single area that covers multiple proposed regions. An attention region of interest may also exclude proposed regions where the probability of detecting an object of interest is low.

[0043] In step 608, each attention region of interest is assigned to a multiscale image classification neural network based on available system resources. In one embodiment, the multiscale neural network includes multiple image classification neural networks, each of which has a different level of computational complexity. By assigning attention regions of interest in this manner, the system can allocate resources to achieve object detection and classification goals. In some embodiments, the attention regions of interest have a region size and a probability of detecting an object of interest, and resources are allocated to maximize the chance of detecting and classifying an object, efficiently use available resources, and / or achieve other system goals. In step 610, a neural network in the multiscale neural network processes each attention region of interest to detect and / or classify an object. In step 612, the system associates object data with a captured image (or video) of the scene, including defining a region identifying a detected object location and a classification label for the detected object.

[0044] Where applicable, hardware, software, or a combination of hardware and software may be used to implement the various embodiments provided by the present disclosure. In addition, where applicable, without departing from the spirit of the present disclosure, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware, and / or both. Where applicable, without departing from the spirit of the present disclosure, the various hardware components and / or software components set forth herein may be divided into subcomponents comprising software, hardware, or both.

[0045] Software according to the present disclosure, such as non-transitory instructions, program code and / or data, can be stored on one or more non-transitory machine-readable media. It is also contemplated that the software identified herein can be implemented using one or more general-purpose or special-purpose computers and / or networked and / or non-networked computer systems. Where applicable, the ordering of the various steps described herein can be changed, combined into composite steps and / or divided into sub-steps to provide the features described herein.

[0046] The above embodiments illustrate but do not limit the present invention. It should also be understood that many modifications and variations are possible based on the principles of the present invention. Therefore, the scope of the present invention is limited only by the appended claims.

Claims

1. A neural network and classifier selection system, the system comprising: a proposal component configured to receive a first data set associated with a scene, the first data set comprising at least one image of the scene; a multiscale neural network comprising a plurality of neural networks, each neural network trained to receive a region of interest and output an object classification, wherein operation of each of the plurality of neural networks has an associated system resource allocation, the associated system resource allocation comprising memory resources and / or processing resources; and an attention coordinator configured to determine regions of interest in the image for processing by the multiscale neural network and to assign each determined region of interest to one of the plurality of neural networks in the multiscale neural network for object classification based at least in part on available system resources and the associated system resource allocation for each of the plurality of neural networks.

2. The neural network and classifier selection system according to claim 1, further comprising: A plurality of input devices includes a visible spectrum image capture device configured to capture an image of the scene and a second input device configured to generate a second data set associated with the scene.

3. The neural network and classifier selection system according to claim 2, wherein: The second input device includes a radar system, an audio capture system, and / or an infrared imaging system.

4. The neural network and classifier selection system according to claim 1, wherein: The first data set also includes an identification of at least one object region.

5. The neural network and classifier selection system according to claim 1, wherein: The proposal component is further configured to analyze the first data set and identify a plurality of proposed regions of interest in the at least one image of the scene.

6. The neural network and classifier selection system according to claim 5, wherein: The proposal component includes a motion target component configured to identify one or more regions in the at least one image associated with detected motion.

7. The neural network and classifier selection system according to claim 5, wherein: The proposal component includes a random search component configured to search the at least one image and identify one or more regions associated with an object.

8. The neural network and classifier selection system according to claim 1, wherein: Each of the plurality of neural networks is configured to process a region of interest having an input image size, and wherein the input image size is different for each of the plurality of neural networks.

9. The neural network and classifier selection system according to claim 1, wherein: The attention coordinator is further configured to: Based at least in part on optimizing a probability of detecting each object of interest in the image and / or minimizing a number of pixels processed by the multiscale neural network, determining regions of interest in the image and assigning each determined region to one of the plurality of neural networks in the multiscale neural network.

10. The neural network and classifier selection system according to claim 1, wherein: The proposal component also includes a short-term goal memory configured to track objects associated with regions classified by the multi-scale neural network and generate predicted proposals for use by the attention coordinator to determine regions of interest in the image.

11. A neural network and classifier selection method, the method comprising: receiving a first data set associated with a scene, the first data set comprising at least one image of the scene; Providing a multiscale neural network comprising a plurality of neural networks, each neural network trained to receive a region of interest and output an object classification, wherein each of the plurality of neural networks has an associated resource allocation, the associated resource allocation comprising memory resources and / or processing resources; determining a region of interest in the image for processing by a multiscale neural network; and Based at least in part on available system resources and the associated resource allocation for each of the plurality of neural networks, each determined region of interest is assigned to one of the plurality of neural networks in the multiscale neural network for object classification.

12. The neural network and classifier selection method according to claim 11, further comprising: capturing an image of the scene; and A second data set associated with the scene is generated.

13. The neural network and classifier selection method according to claim 12, wherein: The second data set includes radar data, audio data, and / or infrared imaging data associated with the scene.

14. The neural network and classifier selection method according to claim 11, wherein: The first data set also includes an identification of at least one object region.

15. The neural network and classifier selection method of claim 11, further comprising analyzing the first data set and identifying a plurality of proposed regions of interest in the at least one image of the scene.

16. The neural network and classifier selection method of claim 15, further comprising identifying one or more regions in the at least one image associated with the detected motion.

17. The neural network and classifier selection method of claim 15, further comprising searching the at least one image using a random search process and identifying one or more regions associated with the object.

18. The neural network and classifier selection method according to claim 11, wherein: Each of the plurality of neural networks is configured to process a region of interest having an input image size, and wherein the input image size is different for each of the plurality of neural networks.

19. The neural network and classifier selection method according to claim 11, wherein: Determining regions of interest in the image and assigning each determined region to one of the plurality of neural networks in the multiscale neural network is based at least in part on optimizing a probability of detecting each object of interest in the image and / or minimizing a number of pixels processed by the multiscale neural network.

20. The neural network and classifier selection method according to claim 11, wherein: Determining the region of interest also includes tracking, using short-term object memory, a region associated with an object classified by the multi-scale neural network, and generating a predicted proposal for a location of the tracked region in an image of the scene.

Images