Guided Multispectral Inspection

Abstract

An imaging system is provided, including a first imaging system capturing initial sensor data in the form of visible domain data and a second imaging system capturing subsequent sensor data in the form of second domain data, where the initial sensor data and the subsequent sensor data are in different spectral domains. A controller subsystem applies machine learning techniques to the visible domain data to detect at least one region of interest in real time, localize at least one object of interest within the at least one region of interest, generate position data for the at least one object of interest, and autonomously guide a focus position of the second imaging system to a region of a scene containing the object of interest to capture the second domain data in response to the position data.

Description

[Technical Field]

[0001] The present invention relates generally to imaging systems, and more particularly to guided multi-spectral inspection. [Background technology]

[0002] The advent of small and portable image sensors, such as cameras, infrared (IR) cameras, and imaging radar, has made it possible to perform multispectral imaging of a given scene. Each part of the spectrum provides different information that may be relevant to a given application. For example, millimeter wave (mmWave) imaging radar has the potential to obtain information (approximate shape and reflectivity) from objects located behind / inside opaque materials, such as carton packaging or fabric.

[0003] At the same time, computer vision (CV) algorithms and machine-learning (ML) methods have matured significantly and are now capable of automatically extracting information from visible-domain cameras or videos or a combination thereof, for example, automatically identifying the location of certain objects in a scene.

[0004] A millimeter-wave imager works by forming a beam, directing it to illuminate a given part of a scene, receiving the reflected signal (the received beam is pointed in the same direction), and then through signal processing techniques obtaining the reflectivity and distance from objects in the direction of the beam. This process is repeated at multiple locations to form an image.

[0005] There is typically a trade-off between millimeter wave imaging performance (speed, accuracy, signal-to-noise ratio, spatial resolution) and the size of the region of interest: although millimeter wave images can have a wider overall field of view (FoV) than cameras, for best results (higher frame rates, higher spatial resolution, less energy consumption), it is preferable to scan only the most relevant part of the scene at a time. Summary of the Invention [Means for solving the problem]

[0006] According to an aspect of the present invention, there is provided an imaging system comprising a first imaging system that captures initial sensor data in the form of visible domain data, a second imaging system that captures subsequent sensor data in the form of second domain data, where the initial sensor data and the subsequent sensor data are in different spectral domains, and a controller subsystem that applies machine learning techniques to the visible domain data to detect at least one region of interest in real time, localize at least one object of interest within the at least one region of interest, generate position data for the at least one object of interest, and autonomously guide a focus position of the second imaging system to a region of a scene containing the object of interest to capture the second domain data in response to the position data.

[0007] According to another aspect of the present invention, a method for imaging is provided. The method includes capturing initial sensor data in the form of visible domain data by a first imaging system. The method further includes detecting at least one region of interest in real time by a controller subsystem by applying machine learning techniques to the visible domain data. The method also includes localizing at least one object of interest within the at least one region of interest by the controller subsystem to generate position data for the at least one object of interest. The method additionally includes autonomously guiding a focal position of a second imaging system to the same or a similar scene by the controller subsystem to capture subsequent sensor data in the form of second domain data in response to the position data.

[0008] According to yet another aspect of the present invention, a computer program product for imaging is provided. The computer program product includes a non-transitory computer-readable storage medium having program instructions embedded therein. The program instructions are executable by a computing system to cause the computing system to perform a method comprising capturing initial sensor data in the form of visible domain data by a first imaging system of the computing system. The method further includes detecting at least one region of interest in real time by a controller subsystem of the computing system by applying machine learning techniques to the visible domain data. The method also includes localizing at least one object of interest within the at least one region of interest by the controller subsystem and generating position data for the at least one object of interest. The method additionally includes autonomously guiding a focal position of a second imaging system by the controller subsystem to the same or a similar scene and capturing subsequent sensor data in the form of second domain data in response to the position data, wherein the initial sensor data and the subsequent sensor data are in different spectral domains.

[0009] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

[0010] The following description provides details of preferred embodiments with reference to the accompanying drawings. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary processing system according to an embodiment of the present invention. [Figure 2]FIG. 2 is a block diagram illustrating an exemplary artificial neural network (ANN) architecture according to an embodiment of the present invention. [Figure 3] FIG. 3 is a block diagram illustrating an exemplary neuron, according to an embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram illustrating an exemplary system for guided multispectral inspection in accordance with an embodiment of the present invention. [Figure 5] FIG. 5 is a flow diagram illustrating an exemplary method for guided multispectral inspection, according to an embodiment of the present invention. [Figure 6] FIG. 6 is a flow diagram illustrating an exemplary method for guided multispectral inspection, according to an embodiment of the present invention. [Figure 7] FIG. 7 is a block diagram illustrating an exemplary neural network configuration used by the system of FIG. 4, according to an embodiment of the present invention. [Figure 8] FIG. 8 is a block diagram illustrating an exemplary cloud computing environment having one or more cloud computing nodes with which local computing devices used by cloud consumers communicate, according to an embodiment of the present invention. [Figure 9] FIG. 9 is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Embodiments of the present invention are directed to guided multispectral inspection.

[0013] One or more embodiments of the present invention enable information extracted in one spectral domain (e.g., the visible domain) to guide the operation of a sensor in a different spectral domain (e.g., an imaging radar). In particular, one or more embodiments of the present invention provide that the visible domain information defines a specific location within a scene at which the imaging radar or a second imaging device should focus.

[0014] One or more embodiments of the present invention may involve using artificial intelligence (AI) driven attention to identify objects of interest.

[0015] As one example of the problem solved by the present invention, note that there is a general trade-off between millimeter-wave imaging performance (speed, accuracy, signal-to-noise ratio, system complexity) and the size of the region of interest. While a millimeter-wave imager (or exemplary second imaging system) may have a wider overall field of view (FoV) than a camera (or exemplary first imaging system), for best results it is preferable to scan only the most relevant portion of the scene at a given time. One or more embodiments of the present invention address this trade-off by processing images captured using the first imaging system to autonomously detect the region of interest, which can then be rapidly scanned by a millimeter-wave imager.

[0016] Therefore, the present invention uses a pre-trained neural network that has learned to direct attention to regions of interest in images held by the first imaging system to guide a second imaging system to the same or similar scenes.

[0017] In one embodiment, a first imaging system is configured to capture initial sensor data in the form of visible domain data, and a second imaging system is configured to capture subsequent sensor data in the form of second spectral domain data. In one embodiment, the subsequent imaging data has different perceptual characteristics than the initial imaging data. For example, the second imaging data can be from a millimeter-wave radar having 3D perception capabilities and the ability to detect objects partially or completely covered by opaque materials in the visible domain, such as cloth, cardboard, and plastic.

[0018] FIG. 1 is a block diagram illustrating an exemplary processing system 100 according to an embodiment of the present invention. The processing system 100 can function as the sensor control and data processing subsystem 430 of FIG. 4. The processing system 100 includes a set of processing units (e.g., CPU) 101, a set of GPUs 102, a set of memory units 103, a set of communication units 104, and a set of peripheral devices 105. The CPU 101 can be a single-core CPU or a multi-core CPU. The GPU 102 can be a single-core GPU or a multi-core GPU. The one or more memory units 103 can include cache, RAM, ROM, and other memory (flash, optical, magnetic, etc.). The communication units 104 can include wireless communication devices or wired communication devices (e.g., network (e.g., Wi-Fi, etc.) adapters, etc.), or a combination thereof. The peripheral devices 105 can include display devices, user input devices, printers, imaging devices, etc. The elements of processing system 100 are connected by one or more buses or networks (collectively indicated by drawing reference numeral 110).

[0019] In one embodiment, the processing system 100 further includes a controller 177 for guided multispectral inspection. The controller 177 can be implemented by an ASIC, an FPGA, or the like. In one embodiment, the controller 177 has on-board memory for storing program code for performing the guided multispectral inspection. In another embodiment, the program code can be stored in the memory device 103. In another embodiment, the controller 177 is embodied by one or more CPUs 101 or one or more GPU(s) 102, or a combination thereof. These and other variations can be readily implemented as will be readily understood by those skilled in the art in view of the teachings of the present invention provided herein.

[0020] In one embodiment, memory device 103 can store specially programmed software modules to transform the computer processing system into a special purpose computer configured to implement various aspects of the present invention. In one embodiment, special purpose hardware (e.g., application specific integrated circuits, field programmable gate arrays (FPGAs), etc.) can be used to implement various aspects of the present invention.

[0021] Of course, processing system 100 may also include other elements (not shown) or omit certain elements, as readily contemplated by those skilled in the art. For example, various other input or output devices, or combinations thereof, may be included in processing system 100, depending on the specific implementation, as readily understood by those skilled in the art. For example, various types of wireless, wired, or combinations thereof input or output devices, or combinations thereof, may be used. Moreover, additional processors, controllers, memories, etc. in various configurations may also be utilized. Furthermore, in other embodiments, cloud configurations may be used (e.g., see FIGS. 8-9). These and other variations of processing system 100 are readily contemplated by those skilled in the art in view of the teachings of the present invention provided herein.

[0022] Additionally, the various figures, as attached hereto, should be understood with respect to the various elements and processes associated with the present invention that may be implemented in whole or in part by one or more elements of system 100.

[0023] As used herein, the term "hardware processor subsystem" or "hardware processor" can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, or a separate processor or a computing element-based controller (e.g., logic gates, etc.), or a combination thereof. The hardware processor subsystem can include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, the hardware processor subsystem can include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.), which can be on-board, off-board, or dedicated for use by the hardware processor subsystem.

[0024] In some embodiments, the hardware processor subsystem comprises and is capable of executing one or more software elements, which may include an operating system, one or more applications, or specific code, or any combination thereof, to achieve a specified result.

[0025] In other embodiments, the hardware processor subsystem may comprise specialized circuitry dedicated to performing one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), FPGAs, or PLAs, or a combination thereof.

[0026] These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0027] Such a hardware processor can be used to perform guided multispectral inspection using multiple sensor data.

[0028] These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0029] 2 is a block diagram illustrating an exemplary artificial neural network (ANN) architecture in accordance with an embodiment of the present invention. It should be understood that this architecture is purely exemplary and that other architectures or types of neural networks may be used instead. In particular, while a hardware implementation of an ANN is described herein, it should be understood that the neural network architecture may be implemented or simulated in software. The hardware implementation described herein is included with the intent of illustrating the general principles of neural network computation at a high level of generality and should not be construed as limiting in any way.

[0030] Moreover, the layers of neurons and the weights connecting them described below are described in a general manner and can be replaced by any type of neural network layer having any appropriate degree or type of interconnectivity. For example, the layers can include convolutional layers, pooling layers, fully connected layers, softmax layers, or any other appropriate type of neural network layer. Moreover, layers can be added or removed as needed, and weights can be omitted in favor of more complex forms of interconnection.

[0031] During feedforward operation, a set of input neurons 202 each provide an input voltage in parallel to a respective column of weights 204. In the hardware implementation described herein, each weight 204 has a configurable resistance value, so that a current output flows from the weight 204 to each hidden neuron 206, representing the weighted input. In a software implementation, the weights 204 can simply be represented as a coefficient value that is multiplied against the associated neuron output.

[0032] According to a hardware implementation, the current output by a given weight 204 is determined as follows:

number

[0033] Hidden neurons 206 perform some calculations using the currents from the array of weights 204 and reference weights 207. The hidden neurons 206 then output their own voltages to another array of weights 204. This array operates in a similar manner, with columns of weights 204 receiving voltages from their individual hidden neurons 206 to produce a weighted current output that sums row-wise and is fed to an output neuron 208.

[0034] It should be understood that any number of these stages can be implemented by interposing additional layers of arrays and hidden neurons 206. It should also be noted that some neurons can be constant neurons 209 that provide constant output to the array. Constant neurons 209 can reside between input neurons 202 or hidden neurons 206, or a combination thereof, and are used only during feedforward operation.

[0035] During backpropagation, output neurons 208 provide voltages back across the array of weights 204. The output layer compares the generated network response to training data and calculates an error. The error is applied to the array as a voltage pulse, where the pulse height or duration, or a combination thereof, is modulated proportionally to the value of the error. In this example, columns of weights 204 receive voltages in parallel from individual output neurons 208 and convert the voltages to currents, thereby summing column-wise to provide inputs to hidden neurons 206. Hidden neurons 206 combine a weighted feedback signal with the derivative of their feedforward calculation and store the error value before outputting the feedback signal voltage to their respective column weights 204. This backpropagation continues throughout the network 200 until all hidden neurons 206 and input neurons 202 have stored error values.

[0036] During weight updating, input neurons 202 and hidden neurons 206 apply a first weight update voltage forward, and output neurons 208 and hidden neurons 206 apply a second weight update voltage backward through network 200. The combination of these voltages produces a state change in each weight 204, causing the weight 204 to assume a new resistance value. In this way, weights 204 can be trained to adapt neural network 200 to errors in its processing. It should be noted that the three modes of operation - feedforward, backpropagation, and weight update - are mutually exclusive.

[0037] As described above, weights 204 can be implemented in software or hardware, for example, using relatively complex weighting circuits or using resistive crosspoint devices. Such resistive devices can have nonlinear switching characteristics that can be used to process data. Weights 204 can belong to a class of devices called resistive processing units (RPUs) because their nonlinear characteristics are used to perform calculations in neural network 200. RPU devices can be implemented using resistive random access memory (RRAM), phase change memory (PCM), programmable metallization cell (PMC) memory, or any other device with nonlinear resistive switching characteristics. Such RPU devices can also be considered memristive systems.

[0038] FIG. 3 is a block diagram illustrating an exemplary neuron 300, in accordance with an embodiment of the present invention. This neuron can represent either an input neuron 202, a hidden neuron 206, or an output neuron 208. It should be noted that FIG. 3 shows components to handle all three phases of operation: feedforward, backpropagation, and weight update. However, because the different phases do not overlap, there will necessarily be some form of control mechanism within neuron 300 to control which components are active. Therefore, it should be understood that there may be switches and other structures, not shown, within neuron 300 to handle switching between multiple modes.

[0039] In feedforward mode, difference block 302 determines the value of the input from the array by comparing it to a reference input. This sets both the magnitude and sign (e.g., + or -) of the input from the array to neuron 300. Block 304 performs a calculation based on the input, and its output is stored in storage 305. It is specifically contemplated that block 304 calculates a nonlinear function and can be implemented as an analog or digital circuit or executed in software. The value determined by function block 304 is converted to a voltage in feedforward generator 306, which applies the voltage to the next array. The signal propagates in this manner through multiple layers of arrays and neurons until it reaches the final output layer of neurons. The input is also applied to the derivative of the nonlinear function in block 308, and the output is stored in memory 309.

[0040] During backpropagation mode, an error signal is generated. The error signal can be generated by output neuron 208 or can be calculated by another unit that accepts input from output neuron 208 and compares the output to a correct output based on training data. Otherwise, if neuron 300 is a hidden neuron 206, neuron 300 receives backpropagation information from the array of weights 204 and compares the received information with a reference signal in difference block 310 to provide a continuously valued, signed error signal. This error signal is multiplied by the derivative of the nonlinear function from the previous feedforward step stored in memory 309 using multiplier 312, and the result is stored in storage 313. The value determined by multiplier 312 is converted in backpropagation generator 314 into a backward propagation voltage pulse proportional to the calculated error, thereby applying the voltage to the previous array. The error signal propagates in this manner by passing through multiple layers and neurons of the array until it reaches the input layer of neurons 202 .

[0041] During weight update mode, after both the forward and backward passes are completed, each weight 204 is updated proportional to the product of the signals that passed through the weight during the forward and backward passes. Update signal generator 316 provides voltage pulses in both directions (note that for input and output neurons, only one direction is available). The shape and amplitude of the pulses from update generator 316 are configured to change the state of weight 204, thus updating the resistance of weight 204.

[0042] In contrast to the forward and backward cycles, it is difficult to implement local, fully parallel weight updates in a two-dimensional crossbar array of resistive processing units, independent of the array size. Vector-vector cross products must be calculated, which may require multiplication operations and incremental weight updates to be performed locally at each crosspoint.

[0043] FIG. 4 is a block diagram illustrating an exemplary system for guided multispectral inspection in accordance with an embodiment of the present invention.

[0044] System 400 includes a first imaging system 410, a second imaging system 420, and a sensor control and data processing subsystem 430. In one embodiment, one or both of first imaging system 410 and second imaging system 420 can include a delivery system for delivering the system to an intended scene having one or more potential objects of interest therein. For example, after the first imaging system locates an area of ​​interest, a delivery system, such as a drone, can deploy the second imaging system for additional scanning of the area of ​​interest. However, other embodiments co-locate the first and second imaging systems and operate them simultaneously to minimize latency. Within a fraction of a second, inferences from the second imaging system regarding the area of ​​interest identified by the first imaging system can be obtained.

[0045] In one embodiment, the sensor control and data processing subsystem 430, the first imaging system 410, and the second imaging system 420 enable wireless communication therebetween. In other embodiments, other types of connections can be used.

[0046] The images from both the first imaging system 410 and the second imaging system 420 share a common set of x,y coordinates of the same or similar scene, shown here as the same scene 450. In one embodiment, the originals in both scenes are the same for quick reference to each other.

[0047] In the example of FIG. 4, the scene can be one of a train platform with a bag with an unknown owner.

[0048] The object of interest (in the above example, the unaccompanied bag) and its location can be identified by computer vision (CV) or machine learning (ML) algorithms applied to the data captured by the first imaging system.

[0049] The shared coordinates are used to control the imaging region to which the second imaging system 420 is pointed, presumably equipped with electron beam scanning capability or other means to illuminate a particular FoV and acquire imaging data.

[0050] In one embodiment, first imaging system 410 includes a camera. In one embodiment, the camera is an RGB camera. In one embodiment, the camera is an infrared (IR) camera. Of course, other types of cameras and imaging devices can be included in first imaging system 410.

[0051] In one embodiment, the second imaging device is a radar imager. In one embodiment, the radar imager is a millimeter wave radar imager with beamforming and beamsteering capabilities. Of course, other types of radar imagers and imaging systems can be used as the second imaging device.

[0052] In one embodiment, an implementation of the present invention is configured to use coordinates of an image captured by the first imaging system 410 to define the effective FoV or control the point of aim of the second imaging system 420. This is achieved by sharing coordinates between the two imaging devices / systems 410 and 420. In a practical implementation, the first imaging system 410 and the second imaging system 420 are equipped with electronic or mechanical control mechanisms to control the effective FoV or point of aim of each system.

[0053] It is envisioned that the first imaging device and the second imaging device are different imaging devices capable of capturing respective images in different domains.

[0054] By way of example, and not limitation, imaging systems in accordance with the present invention can be used in vehicles for autonomous driving, defensive driving, and obstacle avoidance, for robotic control in warehouses or manufacturing (vehicles, machines, processor control systems, etc.) facilities, and in countless other applications readily contemplated by those skilled in the art given the teachings of the present invention provided herein.

[0055] 5-6 are flow diagrams illustrating an exemplary method 500 for guided multispectral inspection, according to an embodiment of the present invention.

[0056] At block 505, a neural network is trained offline (i.e., the connections between multiple neurons are optimized) on a training data set containing predefined types of objects of interest.

[0057] At block 510 , initial imaging data is received in the form of visible-domain imaging data for a scene captured by the first imaging system 410 .

[0058] At block 520, a predefined type of object is detected within the scene captured by the first imaging system 410. In one embodiment, the predefined type of object of interest can be detected by applying machine learning techniques to the visible domain. In one embodiment, a visual attention-based neural network can be used that is trained to equip the neural network with the ability to focus on at least one region of interest.

[0059] At block 530, in response to the detection results from block 520, coordinates of relevant objects in the scene captured by the first imaging system 410 are extracted, which coordinates will be shared at block 540 between images captured by both the first imaging system 410 and the second imaging system 420 (hereinafter "shared coordinates").

[0060] At block 540, the second imaging system 420 receives the shared coordinates, controls the aiming direction (field of view (FoV)) of the second imaging system in response to the shared coordinates, and captures subsequent imaging data for the portion of the scene as the initial imaging data including the relevant object. It will be understood that the same coordinates are used, but depending on the characteristics of the second imaging system and the relative positions between the first and second imaging systems, overlapping regions, non-overlapping regions, smaller regions, sampled regions, etc., may be captured as subsequent imaging data relative to the initial imaging data. In one embodiment, if the shared coordinates are not received, the second imaging system 420 is not activated, thus avoiding any energy consumption associated with unnecessarily activating the second imaging system 420.

[0061] At block 550, an action is selectively performed in response to the subsequent imaging data.

[0062] For example, in one embodiment, an autonomously driven motor vehicle is controlled in response to subsequent imaging data. Such control can include braking, steering, acceleration, etc. Such control can be performed to autonomously drive a car in a safe manner and can also be used for autonomous obstacle avoidance by automatically controlling the vehicle via braking, steering, or acceleration to avoid an impending collision with an object of interest. Further to this purpose, the system can alert a user to an obstacle, allowing the user to take control to avoid the obstacle, or simply notify the user of the obstacle while the vehicle is automatically controlled to avoid the object. The vehicle can be a road vehicle, a vehicle used in a warehouse or manufacturing facility (e.g., a pallet loader, a forklift, a mobile robot, etc.). For this application, it is noteworthy that millimeter-wave imaging systems can see through common visual obstacles, such as fog and smoke. For example, a first imaging system in the visible domain can activate a second imaging system in the millimeter-wave domain and target it at a portion of the smoke to find an obstacle behind the smoke.

[0063] In other embodiments, the first imaging system can be located on a first vehicle (e.g., an unmanned vehicle (UAV) having an RGB camera or an IR camera, or a combination thereof), and the second imaging system can be located within a second vehicle (e.g., having a millimeter-wave radar imaging system). The second vehicle and its onboard second imaging system can be directed at the same or similar scene captured by the first vehicle. In this way, a more capable (second) vehicle platform can be selectively deployed only as needed based on the initial imaging data and its processing.

[0064] Although an example has been described that includes two vehicles, one for each of the first imaging system 410 and the second imaging system 420, in other embodiments, only one of the imaging systems 410 or 420 is mounted on a vehicle, while the other is stationary in one location. In one embodiment, the imaging system that is fixed in position can nevertheless be capable of rotation and full 3D movement to achieve targeted image capture.

[0065] In one embodiment, the target environment can be a train platform, where a fixed or mobile camera is used to capture images of the platform from which location information of a suspect bag can be determined. A millimeter wave scanner can be aimed at the location of the bag to deduce the nature of its contents, for example, the presence of a large metal object.

[0066] In one embodiment, a cloud computing platform can be used to perform processing of the initial imaging data (e.g., to determine whether to initiate a call to a second vehicle at all), and then perform position calculations, etc., to provide position data to a specific controller responsible for controlling the FoV or aiming direction of the second imaging system to capture a region of interest within the imaging data captured by the first imaging system. To that end, any of a variety of different types of cloud computing platforms can be used to perform at least some of the steps of method 500, as described in more detail herein below with respect to Figures 7 and 8.

[0067] It should be understood that the steps of method 500 may be repeated for each image or several images captured by first imaging system 410 per block 510 .

[0068] Further description will now be given below of block 520 of method 500 according to an embodiment of the present invention.

[0069] Object recognition is a computer vision method that aims to identify classes of objects in an image and localize them. Object recognition is a trainable deep learning method that can also be pre-trained on a larger dataset. Object recognition can be associated with a segmentation algorithm that localizes a given object and associates it with a bounding box and coordinates. Then, with the right calibration method, the object detector can share the region of interest with the radar.

[0070] Further description will now be given below of block 320 of method 300 according to another embodiment of the present invention.

[0071] Attention mechanisms are machine learning techniques that allow intelligent selection of relevant parts of data (images) given a certain context. Attention mechanisms are parts of neural networks that need to be trained to select specific regions of interest (hard or soft, where soft attention refers to building a map of attention weights and hard attention refers to selecting a precise location). For some applications, the attention mechanism can direct attention to different regions simultaneously or sequentially. Attention mechanisms can be part of a multi-stage inference process that directs attention from one part of an image to another to perform a task. With the right calibration method, the object detector can share regions of interest with radar.

[0072] Figure 7 is a block diagram illustrating an exemplary neural network configuration 700 used by the system 400 of Figure 4, in accordance with an embodiment of the present invention. In particular, the neural network configuration 600 is used by the sensor control and data processing subsystem 430.

[0073] The neural network configuration 700 includes a neural network 710 that has been trained offline (i.e., the connections between neurons have been optimized) on a training dataset that includes a predefined type of object of interest.

[0074] The neural network configuration 700 further includes a visual attention-based neural network 710 that is trained to provide the neural network 720 with the ability to focus on at least one region of interest.

[0075] Neural network 710 is implemented to include a visual attention-based neural network corresponding to a first modality. Neural network 710 is connected to a second modality pipeline 720. In the embodiment of Figure 7, the second modality pipeline is a millimeter wave radar processing pipeline. In other embodiments, other types of modalities can be used for either the first or second modality.

[0076] The neural network 710 receives an input image 711 captured by one or more image sensors. The input image 711 is represented by a matrix having one or more wavelength channels (e.g., RGB, etc.). The input image 711 is resized in a resizing operation 712 and then fed into the neural network 710. The neural network 710 includes at least one 2D convolutional layer (collectively referred to by drawing reference numeral 713), followed by multiple iterations of a 2D pooling layer 714. The neural network 710 further includes a final 2D convolutional layer 715, followed by a fully connected layer 716 that is connected to an output layer 717 for the final result.

[0077] Output layer 717 includes two cells, yaw 717A and pitch 717B, with both positive and negative values ​​to match the equivalent values ​​for a second image sensor, for example a millimeter wave radar imager.

[0078] In conjunction with a millimeter-wave radar image processing pipeline 720, yaw 717A and pitch 717B values ​​from neural network 710 are used to set 721 the center of the effective FoV for the millimeter-wave radar. The millimeter-wave imager operates by scanning a beam across the FoV. A sequence of radar readings 722 is processed using a recurrent neural network as the millimeter-wave radar processing pipeline 720. Thus, the second modality-based neural network 720 includes one or more recurrent layers (collectively indicated by drawing reference numeral 723) and a fully connected layer 724 connected to an output layer 725 for output classification.

[0079] Thus, for a first modality of imagery, a bag on a platform can be detected within / as a first region of interest using a first mode capture device (e.g., an RGB camera) and visual attention. Location information about the region of interest can be used to direct a second mode capture device (e.g., a millimeter wave radar). The output layer can output a classification as to whether the contents of the bag are a potential threat.

[0080] Although this disclosure includes detailed descriptions related to cloud computing, it should be understood that implementation of the teachings recited herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0081] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0082] The features are as follows:

[0083] On-demand self-service: Cloud consumers can unilaterally provision computing capacity, e.g., server time and network storage, as needed, without requiring human interaction with the provider of the service.

[0084] Broad network access: Functionality is available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0085] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, and various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided, but are said to be location-independent in that they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0086] Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, scaled out quickly, released quickly, and scaled in quickly. To the consumer, the capabilities available for provisioning are often unlimited and can be purchased in any quantity at any time.

[0087] Measured Services: Cloud systems automatically control and optimize resource usage by using metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0088] The service model is as follows:

[0089] Software as a Service (SaaS): The capability offered to consumers to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface, such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0090] Platform as a Service (PaaS): The capability offered to consumers to deploy consumer-created or acquired applications, created using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure (e.g., including networks, servers, operating systems, or storage), but does have control over the deployed applications and, in some cases, the application-hosting environment configuration.

[0091] Infrastructure as a Service (IaaS): The capability offered to consumers to provision processing, storage, network, and other basic computing resources on which they can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selecting network components (e.g., host firewalls).

[0092] The deployment models are as follows:

[0093] Private Cloud: Cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by the organization or a third party, and may reside on-premises or off-premises.

[0094] Community Cloud: Cloud infrastructure is shared by several organizations and supports a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party and may reside on-premises or off-premises.

[0095] Public Cloud: Cloud infrastructure is available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0096] Hybrid Cloud: A cloud infrastructure is a blend of two or more clouds (private, community, or public) that remain unique entities but are brought together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0097] A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure comprising a network of interconnected nodes.

[0098] Referring now to FIG. 8 , an exemplary cloud computing environment 850 is illustrated. As shown, the cloud computing environment 850 includes one or more cloud computing nodes 810, with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 854A, a desktop computer 854B, a laptop computer 854C, or an automotive computer system 854N, or combinations thereof, may communicate. The nodes 810 may communicate with each other. The nodes 810 may be physically or virtually grouped into one or more networks (not shown), such as a private cloud, community cloud, public cloud, or hybrid cloud, or combinations thereof, as described herein above. This enables the cloud computing environment 850 to provide infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on their local computing device. It is understood that the types of computing devices 854A-854N shown in FIG. 8 are intended to be illustrative only, and that cloud computing node 810 and cloud computing environment 850 can communicate with any type of computerized device (e.g., using a web browser) over any type of network or network-addressable connection or combination thereof.

[0099] Referring now to Figure 9, one set of functional abstraction layers provided by cloud computing environment 850 (Figure 8) is shown. It should be understood that the components, layers, and functions shown in Figure 9 are intended to be merely exemplary, and that embodiments of the present disclosure are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0100] Hardware and software layer 960 includes hardware and software components. Examples of hardware components include mainframe 961, RISC (Reduced Instruction Set Computer) architecture-based server 962, server 963, blade server 964, storage device 965, and network and networking components 966. In some embodiments, software components include network application server software 967 and database software 968.

[0101] The virtualization layer 970 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 971; virtual storage 972; virtual networks 973, including, for example, virtual private networks; virtual applications and operating systems 974; and virtual clients 975.

[0102] In one example, management layer 980 may provide several functions, as described below. Resource provisioning 981 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 982 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks and protection for data and other resources. User portal 983 provides access to the cloud computing environment for consumers and system administrators. Service level management 984 provides allocation and management of cloud computing resources to ensure required service levels are met. Service level agreement (SLA) planning and fulfillment 985 provides pre-provisioning and procurement of cloud computing resources where future requirements are predicted according to SLAs.

[0103] The workload layer 990 provides examples of functions for which the cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include one of: mapping and navigation 991; software development and lifecycle management 992; virtual classroom instruction delivery 993; data analytics processing 994; transaction processing 995; and guided multispectral inspection 996.

[0104] The present invention may be a system, method, computer program product, or computer program, or any combination thereof, at any level of technical detail that may be integrated. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0105] The computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a ridge structure in a groove in which instructions are recorded, or any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0106] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to an individual computing device / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to the individual computing device / processing device for storage in a computer-readable storage medium.

[0107] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, e.g., Smalltalk, C++, etc., or conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any kind of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0108] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / acts specified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof, to produce a machine. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer-programmable data processing apparatus or other device, or a combination thereof, to function in a particular manner, such that a computer-readable storage medium having stored instructions includes an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof.

[0110] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable data processing apparatus, or other device, implement the functions / acts identified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process.

[0111] Reference herein to "one embodiment" or "embodiment" and other variations of the present invention means that the particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" and other variations in various places throughout this specification do not necessarily all refer to the same embodiment. However, it should be understood that features of more than one embodiment can be combined in light of the teachings of the present invention provided herein.

[0112] It should be understood that the use of any of " / ", "and / or", "at least one of", e.g., "A / B", "A and / or B", "at least one of A and B" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, "A, B and / or C" and "at least one of A, B and C" are intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second alternatives (A and B), the selection of only the first and third alternatives (A and C), the selection of only the second and third alternatives (B and C), or the selection of all three alternatives (A, B and C). This may be expanded as many times as the number of items listed.

[0113] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be accomplished as a single step performed simultaneously, substantially simultaneously, partially, or fully in a time-overlapping manner, depending on the functionality involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, and combinations of multiple blocks in the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special-purpose hardware and computer instructions.

[0114] While preferred embodiments of the system and method have been described, which are illustrative and not intended to be limiting, it should be noted that modifications and variations can be made by those skilled in the art in light of the teachings set forth above. It is therefore to be understood that changes can be made in the particular embodiments disclosed which are within the scope of the present invention as broadly set forth by the appended claims. Accordingly, having described aspects of the present invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. An imaging system, comprising: a first imaging system that captures initial sensor data in the form of visible domain data; a second imaging system that captures subsequent sensor data in the form of second domain data, wherein the initial sensor data and the subsequent sensor data are in different spectral domains; and a controller subsystem operatively connected to the first imaging system and the second imaging system, wherein the controller subsystem detects at least one region of interest in real time by applying machine learning techniques to the visible domain data captured by the first imaging system, localizes at least one object of interest within the at least one region of interest, generates position data for the at least one object of interest, and, in response to the position data, autonomously guides a focus position of the second imaging system to a region of the scene containing the object of interest to capture the second domain data; It is equipped with wherein the first and second imaging systems are configured to share a common set of x and y coordinates for any scene captured thereby, wherein the common set is used to generate the position data, and wherein the generated position data is used to control a field of view (FoV) of the second imaging system. The imaging system.

2. 1. An imaging system, comprising: a first imaging system that captures initial sensor data in the form of visible domain data; a second imaging system that captures subsequent sensor data in the form of second domain data, wherein the initial sensor data and the subsequent sensor data are in different spectral domains; and a controller subsystem operatively connected to the first imaging system and the second imaging system, wherein the controller subsystem detects at least one region of interest in real time by applying machine learning techniques to the visible domain data captured by the first imaging system, localizes at least one object of interest within the at least one region of interest, generates position data for the at least one object of interest, and, in response to the position data, autonomously guides a focus position of the second imaging system to a region of the scene containing the object of interest to capture the second domain data; It is equipped with wherein the first imaging system is disposed on a first vehicle and the second imaging system is disposed on a second vehicle, and the second imaging system is selectively used to capture the second domain data. The imaging system.

3. The imaging system of claim 1 or 2, further comprising autonomously providing depth information and material properties of the at least one object.

4. The imaging system of any one of claims 1 to 3, wherein the first imaging system comprises an imaging device selected from the group consisting of an RGB camera, an infrared (IR) camera, and a thermal camera.

5. The imaging system of claim 4 , wherein the second imaging system comprises a radar imaging system or a millimeter wave imaging system.

6. The imaging system of any one of claims 1 to 5, wherein localizing the at least one object of interest comprises extracting coordinates of the at least one object of interest.

7. The imaging system of any preceding claim, wherein the second domain data comprises information from the scene that is not observable in the visible domain.

8. The imaging system of any one of claims 1 to 7, further comprising controlling an autonomously driven motorized vehicle in response to subsequent imaging data.

9. 10. The imaging system of claim 1, wherein the first imaging system is disposed on a first vehicle and the second imaging system is disposed on a second vehicle, and the second imaging system is selectively used to capture the second domain data.

10. 10. The imaging system of claim 2 or 9, wherein the first vehicle is an unmanned vehicle (UAV), and the first imaging system comprises an imaging device selected from the group consisting of an RGB camera, an infrared (IR) camera, and a thermal camera.

11. 11. The imaging system of claim 1, wherein the first imaging system is a thermal camera for detecting temperature hotspots in a scene as the at least one region of interest.

12. The imaging system of claim 1 , wherein the controller subsystem is configured in a cloud computing configuration.

13. 1. A method for imaging, the method comprising: a first imaging system capturing initial sensor data in the form of visible domain data; a controller subsystem detecting at least one region of interest in real time by applying machine learning techniques to the visible domain data captured by the first imaging system; the controller subsystem localizing at least one object of interest within the at least one region of interest to generate position data for the at least one object of interest; and the controller subsystem, in response to the position data, autonomously directing a focal position of a second imaging system to an area of ​​the scene containing the object of interest to capture subsequent sensor data in the form of second domain data, wherein the initial sensor data and the subsequent sensor data are in different spectral domains; Including, wherein the first and second imaging systems are configured to share a common set of x and y coordinates for any scene captured thereby, wherein the common set is used to generate the position data, and wherein the generated position data is used to control a field of view (FoV) of the second imaging system. The method.

14. 1. A method for imaging, the method comprising: a first imaging system capturing initial sensor data in the form of visible domain data; a controller subsystem detecting at least one region of interest in real time by applying machine learning techniques to the visible domain data captured by the first imaging system; the controller subsystem localizing at least one object of interest within the at least one region of interest to generate position data for the at least one object of interest; and the controller subsystem, in response to the position data, autonomously directing a focal position of a second imaging system to an area of ​​the scene containing the object of interest, such that the second imaging system captures subsequent sensor data in the form of second domain data, wherein the initial sensor data and the subsequent sensor data are in different spectral domains; Including, wherein the first imaging system is disposed on a first vehicle and the second imaging system is disposed on a second vehicle, and the second imaging system is selectively used to capture the second domain data. The method.

15. 15. The method of claim 13 or 14, wherein the second domain data comprises information from the scene that is not observable in the visible domain.

16. The method of any one of claims 13 to 15, further comprising autonomously providing depth information and material properties of said at least one object.

17. 15. The method of claim 14, wherein the first and second imaging systems are configured to share a common set of x and y coordinates for any scene captured thereby, and wherein the common set is used to generate the position data.

18. The method of any one of claims 13 to 17, wherein localizing the at least one object of interest comprises extracting coordinates of the at least one object of interest.

19. The method of any one of claims 13 to 18, further comprising controlling an autonomously driven motorized vehicle in response to the subsequent imaging data.

20. 1. A computer program for imaging, the computer program comprising: detecting, in real time by a controller subsystem of the computing system, at least one region of interest by applying machine learning techniques to visible domain data, wherein the visible domain data is obtained from initial sensor data captured by a first imaging system; localizing at least one object of interest within the at least one region of interest by the controller subsystem to generate position data for the at least one object of interest; and and autonomously directing a focal position of a second imaging system to a region of the scene containing the object of interest in response to the position data to capture subsequent sensor data in the form of second domain data, wherein the initial sensor data and the subsequent sensor data are in different spectral domains. causing one or more processors to execute each step of the method, including wherein the first and second imaging systems are configured to share a common set of x and y coordinates for any scene captured thereby, wherein the common set is used to generate the position data, and wherein the generated position data is used to control a field of view (FoV) of the second imaging system. The computer program.

21. 1. A computer program for imaging, the computer program comprising: detecting, in real time by a controller subsystem of the computing system, at least one region of interest by applying machine learning techniques to visible domain data, wherein the visible domain data is obtained from initial sensor data captured by a first imaging system; localizing at least one object of interest within the at least one region of interest by the controller subsystem to generate position data for the at least one object of interest; and and autonomously directing a focal position of a second imaging system to a region of the scene containing the object of interest in response to the position data to capture subsequent sensor data in the form of second domain data, wherein the initial sensor data and the subsequent sensor data are in different spectral domains. causing one or more processors to execute each step of the method, including wherein the first imaging system is disposed on a first vehicle and the second imaging system is disposed on a second vehicle, and the second imaging system is selectively used to capture the second domain data. The computer program.

Images