Computer-implemented method of detecting a suspicious object and security system using the same

Abstract

A method of detecting a suspicious object, comprising; receiving video image frames S02, removing moving foreground objects in each image frame S03, classifying one or more static objects as suspicious using an object recognition model S04, and generating an output image highlighting the suspicious objects S05. Objects classified as suspicious in different frames are mapped to be the same object if their positions in different frames are not changed and they may be tracked to determine a duration of unattended time, generating an alert if the time exceeds a predetermined threshold. The suspicious objects are selected from a group at least comprising prohibited items, suitcases, handbags and luggage. Removing one or more moving foreground objects may comprise using a pixel-based Adaptive Gaussian Mixture Model (AGMM) and the classifier may be trained using real or artificially generated images. The object recognition model may comprise a trained convolutional neural network.

Description

FIELD OF THE DISCLOSURE The present disclosure relates to security monitoring and, in particular, to the automated identification of suspicious objects. BACKGROUND OF THE DISCLOSURE Public safety is one of the enabling and critical success factors for building resilient cities, while the adoption of Innovation and Technology under smart city initiatives is particularly important to create the ability to rapidly adapt to changes and disturbance. Railway stations are not only critical infrastructure, but also busy public areas with thousands of passengers in a station at a time, or even more in major interchanging stations. Therefore, it is vital to explore effective measures against serious threats to public safety. To ensure railway safety, which is also an important element of corporate responsibility and Environmental, Social and Governance (ESG), rail operators have various requirements regarding the items that can be brought when travelling, for example dangerous or over-sized objects are strictly prohibited. From a global transportation and city perspective, safety and security are also equally important. Terrorism has always been a major threat to public safety. Explosive devices, flammable chemicals, poisonous or otherwise harmful gases may be hidden in unattended luggage and triggered remotely to endanger passers-by. Therefore, identifying suspicious unattended objects early could enable people to take actions proactively and prevent possible casualties. According to some statistics, the global expenditure on anti-terrorism reached $1365 billion in 2017. Despite these efforts, terrorism still claimed more than 6000 lives across the world in 2022. One of the common methods that terrorists use to inflict harm is to plant suspicious unattended objects, such as bombs, flammable chemicals, or poisonous gases in crowded places and detonate remotely or by timers, resulting in a high number of casualties. As such, effective and continuous surveillance is essential to ensure public and railway safety. Closed-Circuit Television (CCTV) cameras are installed in many stations to cover major areas. Human staff members need to monitor the various CCTV camera video streams to spot and identify any possible suspicious objects left unattended. However, there may often be too many cameras to be monitored, which is a major pain point and a challenge for station staff to timely spot suspicious objects, especially when the station is crowded. Conventional object detection methods have relied on humans monitoring surveillance footages or involved traditional Al solutions. Neither was sufficient for crowded environments such as railway stations and airports. The invention addresses the pain points of the existing, delivering improved performance and robustness in detecting suspicious unattended objects in busy environments. Although there are Artificial Intelligence (Al) products in the market that have object detection and unattended object detection embedded, they are far from sufficient due to various constraints such as processing capability and detectable object type. Many such products only work well with still backgrounds, only detect very limited objects with busy background, and easily lose tracking when being occluded by foreground. Figs. 1A to IC illustrate the limitations of existing detection methodologies. For example, as shown in Fig. 1A, backpacks and suitcases on the floor may not be detected, while as shown in Fig. IB, attended items such as bags carried by people may be detected. Moreover, even if an object is detected in the busy environment, traditional approaches may not be able to correlate how long the object has been present in the environment. For example, as shown in Fig. IC, if an object is detected but is occluded at a later time, existing methods may be unable to determine the time of that object in the environment as a timer will be reset. It is an object of the present disclosure to address or at least partially ameliorate some of the above problems of the current approaches. SUMMARY OF THE DISCLOSURE Features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. In accordance with a first aspect of the present disclosure, there is provided a computer-implemented method of detecting a suspicious object. The method comprises: receiving an input stream of image frames; processing a plurality of consecutive image frames to remove one or more moving foreground objects in each image frame; classifying one or more static objects as suspicious using an object recognition model, where the object recognition model is trained using a corpus of images of objects considered to be suspicious; and generating an image for output highlighting the one or more classified suspicious objects. Optionally, the corpus of images of objects for training may comprise one or more real images of suspicious objects. Alternatively or additionally, the corpus of images of objects for training may comprise one or more generated images of suspicious objects. Optionally, removing one or more moving foreground objects in each image frame comprises adopting a pixel-based Adaptive Gaussian Mixture Model (AGMM) to obtain foreground removed images. The object recognition model may comprise a trained convolutional neural network. Optionally, a plurality of objects classified as suspicious in different frames are mapped to be the same object if their positions in different frames are not changed. The method may further comprise tracking the static objects classified as suspicious to determine a duration of unattended time and generating an alert if the duration of unattended time exceeds a predetermined threshold. In accordance with a second aspect of the present disclosure, there is provided a security system. The security system may comprise at least one video camera configured to capture an input stream of images frames; a video pre-processing module and a classification server. The video preprocessing module may be configured to receive an input stream of image frames; and process a plurality of consecutive image frames to remove one or more moving foreground objects in each image frame. The classification server may be configured to classify one or more static objects as suspicious using an object recognition model, where the object recognition model is trained using a corpus of images of objects considered to be suspicious; and generate an image for output highlighting the one or more classified suspicious objects. DESCRIPTION OF THE DRAWINGS In order to describe the manner in which the above-recited and other advantages and features of the disclosure can be obtained, a more particular description of the principles briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended Figures. Understanding that these Figures depict only exemplary embodiments of the disclosure and are not therefore to be considered to be limiting of its scope, the principles herein are described and explained with additional specificity and detail through the use of the accompanying Figures. Preferred embodiments of the present disclosure will be explained in further detail below by way of examples and with reference to the accompanying Figures, in which:- Figs. 1A to IC illustrate exemplary existing detection methodologies. Fig. 2 shows a schematic diagram of a security system, according to an embodiment. Fig. 3 shows an exemplary user interface for a security system. Figs. 4A and 4B show a foreground removal result. Fig. 5 shows an output user interface according to an embodiment. Figs. 6A to Fig. 6C show several scenarios where the security system 10 may be implemented to enhance object recognition. Fig. 7 shows an exemplary training sample. Fig. 8 shows details of a model training process. Fig. 9 show a flowchart of a method of detecting a suspicious object, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without departing from the scope of the disclosure. Referring to the drawings, there is shown in Fig. 2 a security system, according to an embodiment. The security system 10 comprises at least one video camera 100, a video pre-processing module 200 and a classification sever 300. The security system may be installed at a location of interest, for example, a railway station, airport, other mass transit premises or any other public location, e.g. libraries, museums, open public spaces etc. The at least one video camera 100 is configured to capture an input stream of images frames. The at least one video camera 100 may be installed at the location(s) of interest to capture real-time footage of the environment. Fig. 3 shows an exemplary user interface for a security system. In the example of Fig. 3, at least four cameras installed at the desirable locations are used to provide streams of images frames. A larger or smaller number of cameras may be installed, according to the location of interest, based on the size, business, expected risk, acceptable cost and other considerations. The video pre-processing module 200 is configured to receive an input stream of image frames. The video pre-processing module 200 may receive the input stream of image frames from the at least one video camera 100. For example, the input stream of images frames captured by the at least one video camera 100 may be received at the video pre-processing module 200 through network connectivity via common open protocols in real time. The video pre-processing module 200 is configured to process a plurality of consecutive image frames to remove one or more moving foreground objects in each image frame. For example, each image frame may be processed with one or more preceding image frames and / or one or more following image frames. The video pre-processing module 200 may conduct foreground filtering so as to filter noisy foreground occlusions, such as moving foreground objects. During the foreground filtering, foreground is detected and subtracted such that only the background and static objects are remained for fine detection. The moving foreground is therefore filtered. In an example, removing one or more moving foreground objects in each image frame may comprise adopting a pixel-based Adaptive Gaussian Mixture Model (AGMM) to obtain foreground removed images. The AGMM may facilitate the changes in lighting condition. Therefore, the present video pre-processing module 200 may be adapted to indoor, semi- outdoor or outdoor plots. The AGMM may be configured to compare each image frame with up to 500 preceding frames to identify and remove moving foreground objects. Fig. 4A illustrates an image captured by the at least one video camera 100 and Fig. 4B shows the results after the application of foreground detection / subtraction. With the foreground detected, it is possible to filter out temporary moving occlusive objects in a busy environment. It should be understood that other background modelling algorithms can also be employed to achieve similar functionality. The classification server 300 is configured to classify one or more static objects as suspicious using an object recognition model. The object recognition model may be trained for detecting objects on the foreground filtered images. The object recognition model may be any suitable object detection model, for example, a convolutional neural network (e.g. YOLO, RetinaNet), feed-forward convolutional neural network (e.g. single shot detector, SSD), residual neural network (e.g. ResNet50), or region-based convolutional neural network (e.g. Faster, Cascade, Libra). In some examples, a YOLOv5 convolutional neural network may be used. In some examples, the last layer of the convolutional neural network may be dropped. The object recognition model is trained using a corpus of images of objects considered to be suspicious. The corpus of images of objects for training may comprise one or more real images of suspicious objects. For example, a dataset of training images may be collected from railway environment scenarios, including objects with occlusions and unattended objects (for example, objects without people next to it). The classification server 300 is configured to generate an image for output. One or more classified suspicious objects are highlighted in the output image. Fig. 5 shows an output user interface according to an embodiment. As shown in Fig. 5, on an output image stream, suspicious objects may be identified by a red bounding box and / or label, such as a suspicious object detected near a column. The disclosed object recognition model introduces an innovative approach to suspicious object detection in busy railway environments. By automating the detection process, the system eliminates errors caused by human factors and outdated Al solutions. It utilizes advanced computer vision techniques and deep learning algorithms to accurately identify potential threats such as prohibited items and unattended objects from CCTV footage. The system's background modelling techniques filter out noisy foregrounds for better object tracking. Compared with an approach of subtract background from an image to obtain a foreground mask to achieve foreground detection, the present foreground filtering can provide a better detection of static objects. Real-time alarms and video management systems can ensure prompt alerts and enable efficient response by station operators. The system's adaptability allows for potential expansion into other bustling areas and industries, making it a valuable tool in enhancing safety and security in public spaces.) Fig. 6A to Fig. 6C show several scenarios where the security system 10 may be implemented to enhance object recognition. The security system 10 may be configured to recognise luggage without people next to it (as shown in Fig. 6A), filter bags carried by people (as shown in Fig. 6B). Even when objects are occluded (as shown in Fig. 6C), the security system 10 is able to continue monitoring them, i.e., count-up clock would not be reset but would continue. The corpus of images of objects for training may comprise one or more generated images of suspicious objects. For example, training data may be created with 3D modelling software and / or generative Al. In this way, it is possible obtain large quantities of object images which may be otherwise difficult to obtain from real-world scenarios. In one example, both real object images and generated object images may be used. This may at least provide sufficient data for the model training and improve object recognition performance for more relevant objects. Fig. 7 shows an image with backpacks and handbags artificially created by using 3D modelling software. In this example, Blender is used. It should be understood that other similar 3D modelling software is also applicable for generating object images. In this way, the system's training dataset, comprising real and generated images, can further enhance detection accuracy. A convolutional neural network (CNN) may be adopted for the object recognition model, as discussed below in detail. The artificial intelligence framework may be Linux-based, running Python programming language. In one example, it may be running a Linux (Ubuntu) platform, adopting Python as the key programming language. Relevant Python modules may also be applied. For example, an OpenCV Python module may be adopted so as to provide rich computer vision capability and enable efficient video capturing and image pre-processing. The OpenCV Python module forms the foundation of the artificial intelligence framework. In one example, Real-Time Streaming Protocol (RTSP) is adopted for real-time streaming media transmission over computer networks, including receiving input streams of images frames from the at least one video camera 100 and outputting output stream of images to a user interface. The foreground filtering may be conducted on consecutive image frames to remove one or more moving foreground objects in each image frame. After the object detection / recognition is conducted, the overlaid video may be output to user interface as an RTSP out video stream. A PyTorch Python module may be also adopted to conduct inference to images pre-processed by OpenCV and perform object detection and time calculation. PyTorch can conduct inference to the pre-processed image and detect any suspicious objects. Pre-classified types of objects (such as dangerous objects) may be recognised and alarms may be generated. Unattended objects may be separately considered as a category of suspicious object. If the object exists for more than a pre-determined time, alarms may be generated. Considering moving occlusive objects in busy environment, it may also be determined whether an object already exists previously even though it was occluded. In this situation, the time elapsed may be accumulated. A region of Interest (ROI) in each camera view may be predetermined such that only suspicious results within the ROI will be considered and output. This enables efficient, accurate and robust recognition. The suspicious objects may include prohibited items, suitcases, handbags, luggage and the like. Fig. 8 shows details of a model training process of the object recognition model. In one example, the training of the object recognition model may be started with the pre-trained weights. The object recognition model may be further trained over 300 epochs using PyTorch. Around 7000 images may be used for training, with default data augmentation applied. A portion (e.g. 7%) of the training data may be reserved for validation to prevent over-fitting. The weight with best Mean Average Precision (mAP) may be picked. If an object was stationary for a pre-defined period of time, an alarm may be triggered. For unattended objects, if a duration of unattended time exceeds a predetermined threshold, an alert may be generated. A plurality of objects classified as suspicious in different frames may be mapped to be the same object if their positions in different frames are not changed. Fig. 8 shows the Mean Average Precision (mAP) against epochs at different I0U thresholds. mAP @0.5 means IOU>0.5 is counted as a correct case, while mAP@0.5:0.95 means the average of @0.5, @0.55,...@0.95. In the example shown, the mAP peaked at about the 150th epoch and further training reduced the mAP, which means over-fitting has occurred and the training shall stop. The value of mAP highly depends on the diversification of the dataset, a normal value should be above 0.5 for mAP@0.5. The minimal detectable / recognisable object, in this example is about 30px x 30px on screen. It should be understood that other training settings may be also be used to provide a similar result. The trained model has been tested in a real environment of railway station. An accuracy of over 90% was achieved in 100 suspicious object detection tests, and it could deliver a true rate exceeding 90% over a 3-month test period. Overall, it could achieve an Fl score of 0.9, and alarms could be triggered within 1 second upon receiving corresponding video frames. A frontend system may be set to enable users to view results and receive alarms. The frontend system may be configured to stream the RTSP feed from the security system 10 and display it in real-time on a user interface. Additionally, any alarms generated by the security system 10 may be transmitted to the frontend system. Users may be promptly notified through visual and audible cues such that their attention may be drawn for immediate follow-up actions. Human factors are also considered when designing the workflow and user interface to ensure effective operation. Fig. 9 is a flowchart showing a method of detecting a suspicious object, according to an embodiment. The method starts at step SOI. At step S02, an input stream of image frames is received. At step S03, a plurality of consecutive image frames is processed to remove one or more moving foreground objects in each image frame. At step S04, one or more static objects are classified as suspicious by using an object recognition model. The object recognition model may be trained by using a corpus of images of objects considered to be suspicious. At step S05, an image for output is generated with the one or more classified suspicious objects highlighted. The method finishes at step S06. The above embodiments are described by way of example only. Many variations are possible without departing from the scope of the disclosure as defined in the appended claims. For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Methods according to the above-described examples can be implemented using computerexecutable instructions that are stored or otherwise available from computer readable media. Such instructions can comprise, for example, instructions and data which cause or otherwise configure a general-purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to described examples include magnetic or optical disks, flash memory, Universal Serial Bus (USB) devices provided with non-volatile memory, networked storage devices, and so on. Devices implementing methods according to these disclosures can comprise hardware, firmware and / or software, and can take any of a variety of form factors. Typical examples of such form factors include laptops, smart phones, small form factor personal computers, personal digital assistants, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example. The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures. Although a variety of examples and other information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements in such examples, as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to examples of structural features and / or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

Claims

1. A computer-implemented method of detecting a suspicious object, comprising:receiving an input stream of image frames;processing a plurality of consecutive image frames to remove one or more moving foreground objects in each image frame;classifying one or more static objects as suspicious using an object recognition model, where the object recognition model is trained using a corpus of images of objects considered to be suspicious; andgenerating an image for output highlighting the one or more classified suspicious objects.

2. A computer-implemented method of claim 1, wherein removing one or more moving foreground objects in each image frame comprises adopting a pixel-based Adaptive Gaussian Mixture Model (AGMM) to obtain foreground removed images.

3. A computer-implemented method of claim 1 or claim 2, wherein the corpus of images of objects for training comprises one or more real images of suspicious objects.

4. A computer-implemented method of any preceding claim, wherein the corpus of images of objects for training comprises one or more generated images of suspicious objects.

5. A computer-implemented method of any preceding claim, wherein the object recognition model comprises a trained convolutional neural network.

6. A computer-implemented method of any preceding claim, wherein a plurality of objects classified as suspicious in different frames are mapped to be the same object if their positions in different frames are not changed.

7. A computer-implemented method of any preceding claim, further comprising tracking the static objects classified as suspicious to determine a duration of unattended time and generating an alert if the duration of unattended time exceeds a predetermined threshold.

8. A computer-implemented method of any preceding claim, wherein the suspicious objects are one or more selected from a group at least comprising prohibited items, suitcases, handbags and luggage.

9. A security system comprising:at least one video camera configured to capture an input stream of images frames;a video pre-processing module configured to:receive an input stream of image frames;process a plurality of consecutive image frames to remove one or more moving foreground objects in each image frame; anda classification server configured to:classify one or more static objects as suspicious using an object recognition model, where the object recognition model is trained using a corpus of images of objects considered to be suspicious; andgenerate an image for output highlighting the one or more classified suspicious objects.

10. A security system of claim 9, wherein removing one or more moving foreground objects in each image frame comprises adopting a pixel-based Adaptive Gaussian Mixture Model (AGMM) to obtain foreground removed images.

11. A security system of claim 9 or claim 10, wherein the corpus of images of objects for training comprises one or more real images of suspicious objects.

12. A security system of any preceding claim, wherein the corpus of images of objects for training comprises one or more generated images of suspicious objects.

13. A security system of any preceding claim, wherein the object recognition model comprises a trained convolutional neural network.

14. A security system of any preceding claim, wherein a plurality of objects classified as suspicious in different frames are mapped to be the same object if their positions in different frames are not changed.

15. A security system of any preceding claim, further comprising tracking the static objects classified as suspicious to determine a duration of unattended time and generating an alert if the duration of unattended time exceeds a predetermined threshold.

16. A security system of any preceding claim, wherein the suspicious objects are one or more selected from a group at least comprising prohibited items, suitcases, handbags and luggage.

Images