Multi-sensor theft / threat detection system for crowd pre-screening
Through a multi-sensor system combining passive millimeter wave, RGB and depth sensors, using image fusion and multi-frame analysis technology, the limitations of passive millimeter wave sensors in population pre-screening are solved, and effective monitoring of multiple mobile targets is achieved and potential threat detection improves detection accuracy and efficiency.
Patent Information
- Application Number
- CN201880094941.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-06-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2038-06-29
AI Technical Summary
Existing passive millimeter wave sensors have limitations in population pre-screening, cannot be used effectively in busy environments, cannot track multiple mobile targets, and have poor detection results.
A multi-sensor system was designed, combining passive millimeter wave sensors, RGB imaging equipment and depth sensors, and through image fusion technology and multi-frame analysis technology, the monitoring of people and potential threat detection is achieved, and the image analysis module is used to extract individual targets. And identify hidden objects, trigger alerts and store relevant information.
Effective monitoring of multiple mobile targets in busy environments and detection of potential threats is achieved, the limitations of passive millimeter wave sensors in pre-screening of populations are overcome, and the accuracy and efficiency of detection are improved.
Smart Images

Figure CN112654897B_ABST
Abstract
Description
Technical Field
[0001] The present invention generally relates to an apparatus, system or method for a multi-modal sensor image data processing method and system. In particular, although not exclusively, the present invention relates to a multi-sensor theft / threat detection system for crowd pre-screening using image fusion technology. Background Art
[0002] Image fusion (IF) is a technique that integrates complementary multimodal, multitemporal and / or multiview information into a new image that will contain information of a quality that cannot be obtained otherwise.
[0003] There are a wide range of applications in IF technology. In astronomy, multi-sensor fusion is used to achieve high spatial and spectral resolution by combining images from two or more sensors: high spatial resolution sensor, IR sensor, and X-ray sensor. Medical imaging uses IF on simultaneous evaluation of CT, MRI, and / or PET images. Military, security, and surveillance applications use multimodal image fusion of visible light with infrared images. Using these techniques, security systems for theft or threat detection can be implemented. Summary of the invention
[0004] The present invention provides an apparatus, system or method for a multi-mode sensor image data processing method and system, which can be used in a multi-sensor theft / threat detection system using image fusion technology for crowd pre-screening. Advantageously, the present invention can provide a new and novel computer vision surveillance method.
[0005] According to a first aspect of the present invention, there is provided a crowd monitoring device, comprising: a multi-sensor system connected to a processing device, wherein the processing device comprises: an image fusion module for receiving transformed images from the multi-sensor system and fusing them into a fused image; an image analysis module for extracting each individual target and identifying any hidden objects on the individual target; and an alarm triggering module for triggering an alarm in the event that a hidden object on the individual target is identified.
[0006] In an embodiment of the first aspect, the multi-sensor system includes a passive millimeter wave sensor.
[0007] In an embodiment of the first aspect, the multi-sensor system further includes an RGB imaging device, a depth sensor, and a thermal sensor.
[0008] In an embodiment of the first aspect, the multi-sensor system is connected to the processing device via a wired connection.
[0009] In an embodiment of the first aspect, the multi-sensor system is connected to the processing device via a wireless connection.
[0010] In an embodiment of the first aspect, the image analysis module is adapted to analyze a series of time lapse fused images generated by the image fusion module in order to track the individual target and identify any hidden objects on the individual target.
[0011] In an embodiment of the first aspect, the alarm triggering module is adapted to store the alarm, the image and the related information to a database server connected to the processing device.
[0012] In an embodiment of the first aspect, the processing device is adapted to be connected to an alarm visualization device displaying alarms, images and information.
[0013] In an embodiment of the first aspect, the image fusion module comprises a hardware or software module adapted to transform the received images and find mutual alignment by solving a correspondence problem in computer vision.
[0014] In an embodiment of the first aspect, the image analysis module is adapted to process the fused image by performing the following steps:
[0015] Extract each individual from multiple targets on the fused image,
[0016] Finding any suspected contraband from the passive millimeter wave sensor and thermal sensor and correlating it with the extracted individual, and
[0017] Truly hidden objects are distinguished by comparing the images generated from the passive millimeter wave sensor and the thermal sensor.
[0018] In an embodiment of the first aspect, the alarm triggering module is adapted to analyze suspicious targets and hidden objects identified from the image analysis module across multiple frames.
[0019] In an embodiment of the first aspect, the alarm triggering module is adapted to confirm suspected contraband of the individual and generate an alarm by accumulating positive detection results from the image analysis module across a plurality of time-delayed frames.
[0020] In an embodiment of the first aspect, the processing device is associated with a database server for storing and retrieving image data.
[0021] In an embodiment of the first aspect, the database server is connected to the alarm visualization device.
[0022] In an embodiment of the first aspect, the database server is adapted to collect and store information of suspicious objects upon an alarm asserted by the processing device.
[0023] In an embodiment of the first aspect, the visualization device is adapted to receive or retrieve the alarm and its corresponding information from a database.
[0024] In an embodiment of the first aspect, the visualization device comprises a display for displaying information.
[0025] According to a second aspect of the present invention, there is provided a method for crowd pre-screening using multi-sensor theft / threat detection, the method comprising the following steps:
[0026] transforming images obtained from multi-sensor systems;
[0027] Fusing the transformed images;
[0028] extracting each individual target using a depth sensor and a thermal sensor, and tracking the target across multiple frames;
[0029] The passive millimeter wave image of each extracted individual target on the image is analyzed across multiple frames, and any hidden objects on the individual target are identified.
[0030] In an embodiment of the second aspect, the method further comprises the steps of triggering an alarm and storing the image and related information to a database.
[0031] In an embodiment of the second aspect, the method further comprises the step of displaying the alert, image and information on a display device.
[0032] In an embodiment of the second aspect, the multi-sensor system includes a passive millimeter wave sensor.
[0033] In an embodiment of the second aspect, the multi-sensor system further includes an RGB imaging device, a depth sensor, and a thermal sensor.
[0034] In an embodiment of the second aspect, the multi-sensor system is connected to the processing device via a wired connection.
[0035] In an embodiment of the second aspect, the multi-sensor system is connected to the processing device via a wireless connection.
[0036] In an embodiment of the second aspect, the method further comprises the step of analyzing a series of time-lapse fused images generated by the image fusion module to track the individual target and identify any hidden objects on the individual target.
[0037] In an embodiment of the second aspect, the method further comprises the step of storing the alert, the image and related information to a database server.
[0038] In an embodiment of the second aspect, the method further comprises the step of connecting to an alarm visualization device to display alarms, images and information.
[0039] In an embodiment of the second aspect, the transforming images step comprises the steps of transforming images received from the multi-sensor system and finding mutual alignment by solving a correspondence problem in computer vision.
[0040] In an embodiment of the second aspect, the analyzing step further comprises the following steps:
[0041] Extract each individual from multiple targets on the fused image,
[0042] Finding any suspected contraband from the passive millimeter wave sensor and thermal sensor and correlating it with the extracted individual, and
[0043] The truly hidden objects are distinguished by comparing the images generated from the passive millimeter wave sensor and the thermal sensor.
[0044] In an embodiment of the second aspect, the method further comprises the step of analyzing suspicious targets and hidden objects identified from the image analysis module across multiple frames.
[0045] In an embodiment of the second aspect, the method further comprises the steps of confirming the individual's suspected contraband by accumulating positive detection results from the image analysis module across a plurality of time-delayed frames and generating an alarm.
[0046] In an embodiment of the second aspect, a database server is adapted to store and retrieve image data and is associated with a processing device connected to the multi-sensor system.
[0047] In an embodiment of the second aspect, the database server is connected to the alarm visualization device.
[0048] In an embodiment of the second aspect, the database server is adapted to collect and store information of suspicious objects once there is an alarm asserted by the processing device.
[0049] In an embodiment of the second aspect, the visualization device is adapted to receive or retrieve the alarm and its corresponding information from a database.
[0050] In an embodiment of the second aspect, the visualization device comprises a display for displaying information. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:
[0052] Figure 1 A schematic diagram showing a crowd monitoring device according to an embodiment of the present invention is shown;
[0053] Figure 2 shows the two image fusion results obtained with and without the transformation; and
[0054] Figure 3 An overall image fusion result based on individual images obtained by multiple image sensors is shown, as well as the result of locating a hidden object hidden by two different identified persons. DETAILED DESCRIPTION
[0055] The inventors have designed through their own research, trials and experiments that millimeter waves are in a frequency band that can penetrate most clothing materials, making it a widely used imaging device for security applications. These imaging devices can be classified as active devices or passive devices.
[0056] Active devices emit millimeter waves and analyze the reflected waves for scanning. This enables a clearer image to be formed, but it infringes on the privacy of the scanned target. On the other hand, passive devices only receive millimeter waves radiated from the human body to produce images, but due to the low energy received by the device, it always results in a blurred image.
[0057] Passive devices may be subject to several limitations. Since the energy received by the device is low, it requires the target to be stationary in front of the device to obtain a clear image. In addition, it can only scan a single target at a time.
[0058] Without wishing to be bound by theory, these limitations may hinder the use of passive millimeter waves in busy passenger traffic environments. Therefore, maintaining a balance between efficient flow and security management of passenger traffic is always a challenge. Therefore, no existing system has been proposed to use passive millimeter waves for crowd pre-screening purposes.
[0059] Multi-view fusion technology refers to a set of images of the same scene taken by the same sensor but from different viewpoints, which are fused to obtain an image with a higher resolution than the resolution normally provided by the sensor or to cover a wider area than the area normally provided by the sensor or to restore a 3D representation of the scene. The typical application of this technology has been used to make 3D movies.
[0060] Multi-temporal IF techniques recognize two different goals. Images of the same scene are acquired at different times to detect and evaluate changes in the scene or to obtain less degraded images of the scene. The former goal is common in medical imaging, especially in the detection of changes in organs and tumors, and in remote sensing for monitoring land or forest exploitation. Acquisition cycles are often months or years. The latter goal requires that different measurements be made closer to each other, usually in seconds, and possibly under different conditions.
[0061] Multimodal cameras utilize the sensing capabilities of various cameras and integrate their information to provide a more comprehensive understanding of the real-world environment. Image fusion is a typical solution to integrate information from different cameras. This requires solving the correspondence problem in computer vision, which is to find a set of points in one image that can be identified as the same points in another image.
[0062] Multimodal or multiple sensor image fusion is the process of combining related information from several images into one image. The final output image can provide more information than any single image and reduce the signal-to-noise ratio. Users can gather useful information without looking at and comparing images from multiple sensors.
[0063] One of the imported image fusion techniques is the pixel level method, which works in the spatial domain or transform domain. Averaging method, maximization method, Brovey method, principal component analysis (PCA), wavelet transform and intensity-hue-saturation (HIS) based methods belong to the pixel level fusion methods.
[0064] A prerequisite for performing pixel-level fusion is that the images have been acquired by similar sensors, where the sensors are in a stereo setup, of the same modality, same or similar field of view (FOV) and resolution.
[0065] In one example, an apparatus for detecting contraband includes at least one two-dimensional array of spatially distributed point sources of millimeter wave radiation arranged to illuminate a field of view. The point sources include, but are not limited to, oscillators oscillating at the same frequency, means for focusing the millimeter wave radiation from the field of view onto a focal plane, and a two-dimensional array of detectors disposed in the focal plane, each of the detectors generating an output signal in response to the millimeter wave radiation from a particular portion of the field of view. The output signals may be provided to means for displaying an image of the field of view, the pixels of the displayed image corresponding to the output signals generated by the elements of the array.
[0066] This detection device is an example of an active mmWave device. However, because the system relies on the mmWave source to focus on the target, it is not suitable for use in crowded environments.
[0067] In an alternative example, a contraband detection system is provided, comprising a first camera having a first field of view, a second camera having a second field of view, and a display station coupled to the first camera and the second camera. The first camera has an output that provides first image data, the first image data representing radiation in a first frequency band from an item in the first field of view. The second field of view at least partially overlaps the first field of view. The second camera has an output that provides second image data, the second image data representing radiation in a second frequency band different from the first frequency band, representing an item in the second field of view. The display station is adapted to receive the first image data and the second image data. The display station includes at least one computer, the at least one computer being programmed to present a display of the items in the first field of view using the first image data selectively superimposed with an indication of at least one item derived from the second image data. However, such an apparatus is not suitable for use in a crowded environment.
[0068] In yet another example embodiment, a device for detecting contraband is provided. The device includes at least one array of spatially distributed point sources of millimeter wave radiation. The distributed point sources are arranged to illuminate a field of view. The device also has a device for focusing the millimeter wave radiation from the field of view onto a focal plane, and an array of detectors disposed in the focal plane. Each detector is adapted to generate an output signal in response to millimeter wave radiation from a particular portion of the field of view. The output signal may be provided to a device for displaying an image of the field of view, the pixels of the displayed image corresponding to the output signals generated by the elements of the array. However, such a device is not suitable for use in a crowded environment.
[0069] Research on enabling millimeter wave tracking and target imaging has remained in active mode and only allows single target screening. The inventors have designed passive millimeter wave imaging devices that can be used for large-scale pre-screening for theft / threat detection. For example, methods or devices using passive millimeter waves can be used to scan multiple moving targets or provide a fully automatic theft / threat alarm system based on the multi-sensor system.
[0070] The system can utilize passive millimeter waves via a multi-sensor system to overcome the limitations of passive millimeter waves in crowd pre-screening for theft / threat detection. In a preferred embodiment of the present invention, a multi-sensor system is provided, which includes multiple imaging sensors and a passive millimeter wave imaging device for scanning multiple moving targets within the sensor field of view. This can be used for surveillance tracking and theft / threat detection.
[0071] In a preferred embodiment of the present invention, the multi-sensor system uses a multi-frame analysis technique in which the system analyzes multiple consecutive frames to eliminate noise caused by moving targets so that the target does not need to be stationary to be scanned.
[0072] In another embodiment of the present invention, a multi-sensor system uses multiple sensors with various capabilities to segment and detect any suspicious hidden objects for each individual in a crowd.
[0073] In this specification, a passive millimeter wave sensor or imaging device is a sensor or device adapted to receive millimeter waves directly emitted or radiated from a body or object rather than receiving millimeter waves reflected from the body or object. It is different from an active millimeter wave sensor or imaging device adapted to receive millimeter waves emitted from an illumination source to the body and reflected back from the body.
[0074] The multi-mode sensor or multi-mode imaging device is a sensor or imaging device for sensing or taking images of non-millimeter wave electromagnetic waves. In one embodiment, the multi-mode sensor or multi-mode imaging device is any one of an RGB camera, a depth camera, a thermal camera, or an infrared sensor.
[0075] In such Figure 1 In one embodiment of the present invention shown, a crowd monitoring device 10 is provided, comprising: a multi-sensor system connected to a processing device 13.
[0076] In this embodiment, the multi-sensor system should include at least one passive millimeter wave imaging device 11 and one or more multi-mode sensors 12, such as Figure 1 RGB camera, depth camera, and thermal camera shown.
[0077] The processing device 13 includes an image fusion module 14, an image analysis module 15, and an alarm triggering module 16. The processing device 13 is adapted to obtain image signals from the multi-sensor system and process the images using different software modules. In one embodiment, the multi-sensor system is directly connected to the processing device 13 via a data cable. In another embodiment, the multi-sensor system is remotely connected to the processing device 13 via a wireless protocol such as WiFi or Bluetooth.
[0078] refer to Figure 1 , the image fusion module 14 comprises a hardware or software module that can transform the images and find mutual alignment by solving the correspondence problem in computer vision.
[0079] The image analysis module 15 is a hardware or software module that processes the fused image to perform the following operations: 1) extract each individual from multiple targets, 2) discover suspected contraband and associate it with the extracted individual, and 3) distinguish true hidden objects to enhance the detection capability of passive millimeter waves.
[0080] The alarm trigger module 16 is a hardware or software module that analyzes suspicious targets and hidden objects identified from the image analysis module across multiple frames. By analyzing a series of time-shifted frames, this module can trigger a more accurate alarm.
[0081] The processing device 13 is associated with a database server 17 for storing and retrieving image data. The database server 17 is connected to an alarm visualization device 18.
[0082] Once the processing device 13 issues an alarm, the database server 17 will collect and store the suspect's detailed information from the processing device. The visualization device 18 is adapted to receive or retrieve the alarm and its corresponding detailed information from the database and display the information on the display of the visualization device (e.g. screen, tablet...).
[0083] In one embodiment of the present invention, a method for crowd pre-screening using multi-sensor theft / threat detection is provided, the method comprising the following steps:
[0084] transforming images obtained from multi-sensor systems;
[0085] Fusing the transformed images;
[0086] extracting each individual target using a depth sensor and a thermal sensor, and tracking the target across multiple frames;
[0087] analyzing a series of time-shifted passive millimeter wave images generated by the passive millimeter wave sensor to identify any hidden objects on the individual target;
[0088] triggering an alarm and storing the image and related information in the database; and
[0089] The alarm, image and said information are displayed on a display device.
[0090] In one embodiment, the multi-sensor system includes a passive millimeter wave sensor 11 , an RGB imaging device 12 , a depth sensor 12 , and a thermal sensor 12 .
[0091] In one embodiment, the multi-sensor system is adapted to perform a calibration process on the imaging sensors of the multi-sensor system. This is done by first calibrating the multi-sensor system to obtain a pixel mapping relationship between images. This is an offline process and only needs to be performed once before the system is set up.
[0092] All images are transformed based on the pixel mapping relationship. This transformation allows them to be aligned on the same camera plane for easy image fusion.
[0093] In a preferred embodiment, the step of fusing the images is performed by matching feature points. In another embodiment, the step of fusing the images is performed by resolving parallax with the aid of a depth sensor. If there are feature points common to all images, the fusing step can be done by matching feature points. If there are no common feature points, the fusing step can be done by modeling the parallax between the cameras and translating the images in real time based on the depth information.
[0094] Figure 2 The difference between image fusion with and without image transformation is shown. The translation vectors obtained from matching feature points or modeling the difference between imaging sensors are used to align the transformed images.
[0095] In the step of extracting the respective targets on the image, the processing device 13 is adapted to analyze the images from the depth sensor and the thermal sensor. The processing device 13 extracts or recognizes the objects from the depth sensor by analyzing the images from the depth sensor in different depth layers.
[0096] The fusion module 14 of the processing device 13 then fuses the images from the depth sensor and the thermal sensor. By fusing the depth and thermal images, the processing device 13 is adapted to determine whether the object extracted from the depth sensor is a human (individual target). This is based on the assumption that the human body has a constant temperature, which is generally distinguishable from other objects.
[0097] After setting the bandwidth of the heat detection area, non-human objects can be further filtered out for further processing.
[0098] In this specification, a hidden object is an object positioned and covered by one or more layers of material without any covering object (e.g., a bag) in front of the material. The material must be a material that can be penetrated by millimeter waves. In other applications, the passive millimeter wave sensor can be replaced by an X-ray sensor so that the material can be thicker and of a different type, such as plastic or a fresh person.
[0099] Once the individual targets are identified, the images are passed to the image analysis module 15 to perform an automatic hidden object detection method based on the images from the passive millimeter wave sensor 11 .
[0100] The above extraction method can segment each individual target. Combining this result with the automatic hidden object detection on the passive millimeter wave image, the processing device 13 can detect each individual in the crowd one by one.
[0101] Figure 3 The hidden object detection method according to the embodiment of the present invention is shown.
[0102] The analysis module 15 is adapted to analyze the passive millimeter wave image to indicate the detection of hidden objects in two cases: 1) the hidden object is only hidden under the clothes, and 2) there is an obstructing object in front of the clothes, but the clothes with the obstructing object in front of the clothes are not very noticeable. One embodiment of the present invention is to detect the hidden object hidden under the clothes without any obstruction protruding in front of the clothes.
[0103] In one embodiment, the analysis module 15 is adapted to distinguish between these two situations by analyzing the thermal image and the passive millimeter wave image. In the case where a covering is found in front of the clothing, the surface temperature is generally lower than the surface temperature of the clothing. Therefore, the thermal image will show a sudden change in temperature on the surface of the clothing. In the case where a hidden object is hidden under the clothing, the surface temperature is relatively more constantly distributed on the surface of the clothing.
[0104] In a preferred embodiment, the analysis module 15 is adapted to distinguish between these two events by analyzing the temperature distribution on the surface of the garment and the steepness of the temperature change.
[0105] In one embodiment, the processing device 13 is adapted to track each individual target and its movement using a computer vision algorithm, such as a Kalmar filter.
[0106] The processing device 13 is adapted to save all positive alarms in a plurality of consecutive frames in an alarm record associated with each tracked individual. The processing device 13 may store the alarm record in its own storage device, such as a hard drive, solid state drive, RAM, etc. In other embodiments, the processing device 13 may store the alarm record in a database server 17.
[0107] In case of discovery of a hidden object, the processing device 13 is adapted to issue an alarm and store the data in the database server 17 when there is at least more than one alarm within each alarm record found for a particular tracked individual.
[0108] In one embodiment, the processing device 13 is associated with an alarm visualization device 18 which is adapted to periodically query the database server 17 and pop up any new alarms and display the suspect's details on the device.
[0109] It will also be understood that where the methods and systems of the present invention are implemented in whole or in part by a computing system, any suitable computing system architecture may be utilized. This will include stand-alone computers, network computers, and dedicated hardware devices. Where the terms "computing system" and "computing device" are used, these terms are intended to cover any suitable arrangement of computer hardware capable of implementing the described functionality.
[0110] Those skilled in the art will appreciate that, without departing from the spirit or scope of the invention as broadly described, many changes and / or modifications may be made to the invention as shown in the specific embodiments. Therefore, the embodiments of the present invention are considered in all aspects to be illustrative and not restrictive.
[0111] Unless otherwise indicated, any reference to prior art contained herein is not to be taken as an admission that the information is common general knowledge.
Claims
1. A crowd monitoring device, comprising: Multi-sensor system, connected to processing equipment, The processing equipment includes: An image fusion module, used for receiving images from the multi-sensor system, transforming and fusing the images into a fused image; An image analysis module for extracting each individual target and identifying any hidden objects on the individual target; an alarm triggering module, for triggering an alarm in the event that a hidden object on the individual target is identified; The multi-sensor system includes a passive millimeter wave sensor, an RGB imaging device, a depth sensor and a thermal sensor, and The image analysis module is adapted to analyze a series of time-lapse fused images generated by the image fusion module to track the individual target and identify any hidden objects on the individual target.
2. Crowd monitoring device according to claim 1, wherein the multi-sensor system is connected to the processing device by a wired connection.
3. The crowd monitoring device of claim 1, wherein the multi-sensor system is connected to the processing device via a wireless connection.
4. The crowd monitoring device according to claim 1, wherein the alarm triggering module is adapted to store the alarm, the image and related information in a database server connected to the processing device.
5. Crowd monitoring device according to claim 4, wherein the processing device is adapted to be connected to an alarm visualization device which displays alarms, images and said information.
6. The crowd monitoring device according to claim 5, wherein the image fusion module comprises: A hardware or software module adapted to transform the received images and find mutual alignment by solving the correspondence problem in computer vision.
7. The crowd monitoring device of claim 6, wherein the image analysis module is adapted to process the fused image by performing the following steps: extracting each individual from the plurality of objects on the fused image; Finding any suspected contraband from the passive millimeter wave sensor and thermal sensor and correlating it with the extracted individual; and The real hidden objects are distinguished by comparing the images generated from the passive millimeter wave sensor and the thermal sensor.
8. The crowd monitoring device according to claim 7, wherein the alarm triggering module is adapted to analyze the suspicious targets and hidden objects identified from the image analysis module across multiple frames.
9. Crowd monitoring device according to claim 8, wherein the processing device is associated with a database server for storing and retrieving image data.
10. Crowd monitoring device according to claim 9, wherein the database server is connected to an alarm visualization device.
11. The crowd monitoring device according to claim 10, wherein: The database server is adapted to collect and store information of suspicious objects once there is an alarm asserted by the processing device.
12. The crowd monitoring device according to claim 11, wherein the visualization device is adapted to receive or retrieve alarms and their corresponding information from the database server.
13. Crowd monitoring device according to claim 12, wherein the visualization device comprises a display for displaying the information.
14. A method for crowd pre-screening using multi-sensor theft / threat detection, the method being performed by a crowd monitoring device comprising a multi-sensor system connected to a processing device, the method comprising the steps of: transforming an image obtained from the multi-sensor system, the multi-sensor system comprising a passive millimeter wave sensor, an RGB imaging device, a depth sensor, and a thermal sensor; Fusing the transformed images; extracting each individual target using the depth sensor and the thermal sensor, and tracking the target across multiple frames; analyzing the passive millimeter wave image of each extracted individual target on the fused transformed image across multiple frames and identifying any hidden objects on the individual target; The extracting step includes the following steps: analyzing a series of time-lapse fused images generated by the fusion step to track the individual target and identify any hidden objects on the individual target.
15. The method according to claim 14, further comprising the steps of: An alarm is triggered and the image and related information are stored to a database server.
16. The method according to claim 15, further comprising the steps of: The alarm, image and said information are displayed on a display device.
17. The method of claim 14, wherein the multi-sensor system is connected to the processing device via a wired connection.
18. The method of claim 14, wherein the multi-sensor system is connected to the processing device via a wireless connection.
19. The method according to claim 16, further comprising the steps of: The alarm, the image and related information are stored in a database server.
20. The method according to claim 19, further comprising the steps of: Connected to an alarm visualization device for displaying alarms, images and said information.
21. The method of claim 20, wherein the step of transforming the image comprises the steps of: The images received from the multi-sensor system are transformed and found to be aligned with each other by solving the correspondence problem in computer vision.
22. The method of claim 21, wherein the analyzing step further comprises the following steps: extracting each individual from the plurality of objects on the fused image; finding any suspected contraband from the passive millimeter wave sensor and thermal sensor and correlating it with the extracted individual; as well as The real hidden objects are distinguished by comparing the images generated from the passive millimeter wave sensor and the thermal sensor.
23. The method according to claim 22, further comprising the steps of: Suspicious targets and hidden objects identified from the image analysis module are analyzed across multiple frames.
24. The method according to claim 23, further comprising the steps of: Individual suspected contraband is identified and an alert is generated by accumulating positive detection results from the image analysis module across multiple time-shifted frames.
25. The method of claim 24, wherein the database server is adapted to store and retrieve image data and is associated with the processing device connected to the multi-sensor system.
26. The method according to claim 25, wherein: The database server is connected to the alarm visualization device.
27. The method according to claim 26, wherein: The database server is adapted to collect and store information of suspicious objects upon an alarm asserted by the processing device.
28. The method according to claim 27, wherein the visualization device is adapted to receive or retrieve alarms and their corresponding information from the database server.
29. The method of claim 28, wherein the visualization device comprises a display for displaying the information.
