System for screening items of baggage for threat items

Abstract

A system for screening items of baggage for threat items The system comprises: a weight sensor, configured to collect data indicative of the weight of an item of baggage and objects within when the item of baggage is in contact with at least a part of the weight sensor; a sensor array, configured to collect data indicative of the item of baggage and objects within the item of baggage; and a processor, configured to receive the data indicative of the weight of the item of baggage and objects within, and the data indicative the item of baggage and objects within the item of baggage collected by the sensor array, and derive a risk estimation from the received data.

Description

[0001] SYSTEM

[0002] TECHNICAL FIELD

[0003] SYSTEM FOR SCREENING ITEMS OF BAGGAGE FOR THREAT ITEMS

[0004] BACKGROUND

[0005] The screening items of baggage in the context of high-footfall venues has classically been dominated by hand searching, and is sometimes augmented or replaced by x-ray scanners and / or metal detection archways or hand-held wands. However, whilst such techniques may be suited to environments where there is relatively low footfall it is difficult to apply them to environments where a large number of individuals will pass through in any given period of time. In these environments, it is typically not acceptable to interrupt the so-called ‘normal flow of commerce’ by introducing significant delays to individuals. Further, the use of x-rays provides a complicated environment for workers with additional safety considerations.

[0006] The present disclosure was arrived at in light of the above considerations.

[0007] SUMMARY

[0008] Accordingly, in a first aspect, embodiments of the invention provide a system for screening items of baggage for threat items, the system comprising: a weight sensor, configured to collect data indicative of the weight of an item of baggage and objects within when the item of baggage is in contact with at least a part of the weight sensor; a sensor array, configured to collect data indicative of the item of baggage and objects within the item of baggage; and a processor, configured to receive the data indicative of the weight of the item of baggage and objects within, and the data indicative of the item of baggage and objects within the item of baggage collected by the sensor array, and derive a risk estimation from the received data.

[0009] Such a system overcomes the difficulties identified above, in that it can quickly and accurately derive a risk estimation, that is ascertain the probability that an item of baggage contains a threat item. Moreover, it can do so without the use of ionising radiation.

[0010] The sensor array may include a plurality of non-contact sensors. That is, sensors which may collect the data without being in physical contact with the item of baggage. In some examples, the sensor array and weight sensor are in a static arrangement relative to one another. In use, an operator would place the item of baggage on the part of the weight sensor and the sensor array would collect the data without the item of baggage moving relative to either the weight sensor or the sensor array.

[0011] The sensor array may be moveable relative to the item of baggage when it is in contact with at least the part of the weight sensor. In some examples, the sensor array may be moveable (e.g. hinged, or on a slidable bearing). In other examples, the system may further include a moveable baggage holder, which in use receives an item of baggage and conveys it along a scanning path, the sensor array being disposed along the scanning path. Such a baggage holder can ensure consistency when scanning items of baggage, by directing the items of baggage along a predefined scanning path. The system may further include a position sensor, configured to sense the position of the item of baggage and / or the position of the baggage holder relative to the sensor array, as it is conveyed along the scanning path. This can allow enhanced timing control over the sensor array, and further improve the consistency with which data is collected. The weight sensor may be integrated into the moveable baggage holder. In some examples, the weight sensor is directed coupled to but does not move with the moveable baggage holder. This can further improve the consistency with which the data is collected. The moveable baggage holder may be manually operable by a user (e.g. a security operative or a patron undertaking self-service). For example, the user may push the moveable baggage holder along the scanning path, or operate a mechanism which causes the same. This manual operation reduces the cost of the system and can increase uptake in usage of the system as it is operated at the user’s own pace. Further the system is safer, as the risk of trapped limbs, fingers, hair or clothing is significantly higher with automated systems. The moveable baggage holder may be driven by a motor or other drive means (such as a gravity driven system) so as to convey the item of baggage along the scanning path.

[0012] The sensor array may be provided in a housing containing a passage through which the item of baggage is conveyed (for example a portal, opening, or gap). In some examples, the sensor array may be provided in a gantry past which the item of baggage is conveyed. By providing a passage, users are guided on their use of the system and so the consistency with which data is collected is enhanced. The passage may be formed by a complete perimeter or a partial perimeter, for example the passage may be defined by a complete ring or an L-shaped portion of the housing, the area encapsulated by the sides of the L-shaped portion defining the passage.

[0013] The system may further include one or more magnetometers, configured to collect data indicative of ferrous metallic items within the item of baggage and provide the data to the processor for the derivation of the risk estimation. The system may further include one or more inductive sensing coils, configured to collect data indicative of ferrous and / or non-ferrous metallic items within the item of baggage and provide the data to the processor for the derivation of the risk estimation. By increasing the dimensionality of the data collected, the accuracy with which the risk estimation can be derived is increased.

[0014] The sensor array may include one or more of: a first radar sensor, configured to collect data indicative of objects within the item of baggage; a second radar sensor, configured to collect data indicative of objects within the item of baggage; and an optical depth sensor (for example a time-of-flight sensor, stereo camera, or projected light sensor), configured to collect data indicative of the size of the item of baggage. The first radar may operate in the mm-wavelength range. The second radar may operate in the microwave wavelength range.

[0015] The processor may be configured to pre-process the received data before deriving the risk estimation.

[0016] Deriving the risk estimation may include extracting one or more features from the received data to derive a feature vector describing the item of baggage. The features comprising one or more of: an estimated weight of the item of baggage; an outer volume of the item of baggage; a volume of one or more objects within the item of baggage; a quantity of metallic objects within the item of baggage; an average depth of the item of baggage as measured from the sensor array; and the dielectric constant of one or more objects within the item of baggage. The feature vector may be derived by a neural network (e.g. by providing raw sensor data to the neural network for determination of important features), or the features may be preselected.

[0017] Deriving the risk estimation may include providing the data indicative of the weight of the item of baggage and the data indicative the item of baggage and objects within the item of baggage received from the sensor array to a pre-trained machine learning algorithm which derives the risk estimation. Deriving the risk estimation may include deriving one or more meta-features from the received data. The one or more meta-features may include one or more of: (i) a density of the item of baggage; (ii) a spatial overlap between signals from induction and magnetometer sensors; (iii) differences in volumes sensed by each of a plurality of sensors; (iv) relative average depths probed within the item of baggage by each of a plurality of sensors; and (v) overlap between the centres of mass of the item of baggage as sensed by each of a plurality of sensors.

[0018] The system may be further configured to obtain background data from the sensor array, and utilise this background data when deriving the risk estimation. For example, the system may be configured to obtain background data from the sensor array when the weight sensor indicates the presence of an item of baggage and before and / or after the item of baggage is scanned by the sensor array. The system may be further configured, during a startup process, to obtain calibration data from an empty moveable baggage holder or test item, and to utilise the calibration data when deriving the risk estimation. In some examples a camera (which may be in the sensor array) may be used to determine there is no item of baggage and trigger the obtaining of the background data. In further examples, the sensor array may continuously collect data and historical data may be used as the background data. Collecting the background data may be triggered by a positional sensor indicating that the movable baggage holder has moved.

[0019] The system may further comprise a camera, configured to obtain an image of the item of baggage. The image may be used, by the processor, to determine the type of baggage. For example, the item of baggage may be classified as a handbag, rucksack, satchel, etc. The image may be presented on a display of the system, for observation by an operator. The image may be annotated with an indication of where in the item of baggage a threat item was detected.

[0020] In a second aspect, embodiments of the invention provide a method of screening items of baggage for threat items, using the system of the first aspect, the method comprising: weighing the item of baggage with the weight sensor; collecting data indicative of the item of baggage and objects within the item of baggage using the sensor array; and deriving the risk estimation using the processor.

[0021] The invention includes the combination of the aspects and preferred features described except where such a combination is clearly impermissible or expressly avoided.

[0022] Further aspects of the present invention provide: a computer program comprising code which, when run on a computer, causes the computer to perform the method of the second aspect; a computer readable medium storing a computer program comprising code which, when run on a computer, causes the computer to perform the method of the second aspect; and a computer system programmed to perform the method of the second aspect.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 shows a system;

[0025] Figure 2 shows a cross-section through the system of Figure 1 ;

[0026] Figures 3A - 3F show various alternative configurations of the system; Figure 4A - 4C show mmW- and microwave-radar images for three example items of baggage;

[0027] Figure 5 illustrates example data from a square and spiral inductance coil respectively as a metallic test object is scanned;

[0028] Figures 6A and 6B show example data from a magnetometer array when scanning a simple point source magnet and a large blade respectively;

[0029] Figure 7 shows a top-down image of an example item of baggage and corresponding volume scan;

[0030] Figure 8 is a plot showing how measurement of weight changes over time;

[0031] Figure 9 is a schematic of data processing steps;

[0032] Figure 10 shows an ROC curve using a random forest classifier;

[0033] Figure 11 shows an ROC curve using an XGBoost classifier;

[0034] Figure 12 shows individual ROC curves for different classes of items of baggage;

[0035] Figure 13 is a confusion matrix for a multiclass classifier;

[0036] Figure 14 shows an ROC curve for a multiclass XGBoost classifier;

[0037] Figure 15 shows an ROC curve using optimised hyperparameters;

[0038] Figure 16 shows features ranked by their contribution to the classification;

[0039] Figure 17 shows an ROC curve for IED and firearms (where bladed weapons were excluded);

[0040] Figure 18 shows an ROC curve where data relating to bladed weapons runs (e.g. axes and generic knives) were removed;

[0041] Figure 19 shows an ROC curve for lEDs only (where firearms and bladed weapons were excluded);

[0042] Figure 20 shows an ROC curve with handguns, bladed weapon runs being removed;

[0043] Figure 21 shows an ROC curve for threat bags weighing more than 15 units, and with no bladed weapons; and

[0044] Figure 22 is a plot showing the accuracy of the classifier broken down by item of baggage class.

[0045] DETAILED DESCRIPTION

[0046] Aspects and embodiments of the present invention will now be discussed with reference to the accompanying figures. Further aspects and embodiments will be apparent to those skilled in the art. Figure 1 shows a system 100 for screening items of baggage. The system includes a weight sensor 101 , located within a table 116, and formed at least in part from a carriage 102. The carriage is keyed so that a corresponding tray 106 slots onto the carriage such that the entire weight of the tray rests on the carriage. The remaining components of the weight sensor are shown in Figure 2, which is a cross-section through the table 116. The carriage 102 rests on load cells 202a and 202b, which in turn fixed to carriage base 204a and 204b. These carriage bases are, in turn, fixed to bearings 206a and 206b which are slidably engaged to monorail 208. A handle 110 is attached to one of the carriage bases and can be used to slide the carriage along the monorail. By attaching the handle to the carriage bases, the user cannot influence the weight sensed by the load cells by application of a force to the handle. The load cells in this example are transducers, and when the weight of an item of baggage presses down on the load cells a resistance wire changes shape causing a change in resistance which can be electrically measured. In this example the load cells were provided as Tedea-Huntleigh model 1042, with four load cells being provided (one at each corner of the carriage) and electrically connected in parallel. The load cells were connected to a load cell amplifier module and data was captured using a programmable logical controller. In this arrangement, the load cells are moved together with the carriage 102 as they are located between the carriage bases 204a and 204b and the carriage itself 102. In other examples, the load cells are fixed relative to the table 116, for example by being located between the monorail 208 and a fixed base. In such examples, the weight sensor can be calibrated to remove the weight of the components of the carriage, carriage base, bearings, and monorail. The advantage of this arrangement is that no moving cable assemblies are required (because the load cells are fixed).

[0047] The system 100 also includes a housing 104, which extends upwards from one side of the table, turns and extends horizontally parallel to the surface of the table. The upper portion of the housing contains a sensor array 103. In this example, the sensor array includes: one or more, preferably a plurality of 60 GHz radar sensors (for example an Acconeer A111 radar sensor); a microwave radar sensor (for example a Walabot 3-10 GHz ultra-wideband impulse radar), and a time-of-flight sensor (for example an ST Microelectronics VL53L5CX). Each of these is described in detail below. The upper portion of the housing also includes a display 108, which in this example is a tablet mounted via a ball mount. The display provides feedback to users (in this instance, a security operative) of the system, for example indicating that an item of baggage has failed or passed the security screening.

[0048] The table 116 is mounted to a base unit 114, which also includes the processor used to control the sensors and analyse the corresponding data. For convenience, the base unit 114 is mounted on a number of casters 112 so that the system can be easily moved. Turning to the millimetre radar sensor, it is a pulse coherent radar (PCR) which transmits short pulses (with a pulse length of the order of 0.1 ns) with a repetition frequency of up to 13 MHz. The duration of these pulses is such that they cover a large proportion of the 57 - 64 GHz band. In this example, 8 radar modules are arranged in a single array perpendicular to the tray scan direction (the longitudinal direction of the table 116, indicated by the arrow). Each module has a pitch of 45 mm, for a total length of the array of 315 mm (which covers the entire tray 106 width). Lenses were placed in front of each module to collimate the radar beams, giving each radar module a beamwidth of around 16°. At the 400 mm gantry height (the upper portion of the housing 104), each module images a region around 100 mm in diameter. The 8 modules were connected in two groups of 4 to two separate subprocessing units which stream the collected data to the processor over an ethernet network.

[0049] The microwaves generated by the microwave radar are significantly longer in wavelength (several cm) than those of the millimetre radar sensor, and so they interact differently with concealed objects and barrier layers. Whilst both radars will generally be able to penetrate through the outside of an item of baggage, the microwave radar is affected less than the millimetre radar by the outer material of the item of baggage. In this example, the microwave radar is a programmable 3D sensor based on a single chip. The measurements from multiple transmit-receive antenna pairs are analysed to reconstruct a three-dimensional “image” of the environment. In this example, a 3-channel version of the microwave radar was used, which provided a frame rate of around 50 frames per second.

[0050] The sensor array 103 also includes a time-of-flight ranging sensor, in this example an ST Microelectronics VL53L5CX multizonal ToF sensor. The ToF sensor is a ranging sensor which integrates a laser and silicon avalanche photodiode array together with the filters and optics required for imaging. The example ToF sensor used provides 8 x 8 sensing zones (64 pixels) over a 63° diagonal field of view.

[0051] The system also includes metal detection sensors 210 integrated into the surface of the table, one of which is shown in the cross-section view in Figure 2. Ferrous and non-ferrous metallic items can be detected using an inductance coil technique, which is first example of the metal detection sensor. The proximity of nearby metallic items causes a change in the electrical properties of a coil of wire in the manner known per se in the art. The changes in the inductance of the sensing coils were recorded using an LDC1612 inductance-to-digital chip. Advantageously, this allows the inductance to be read directly without the use of an intermediate analogue processing step. The chip had two inputs, and so two different coils were provided: a square wound coil and a spiral wound coil, both planar. The coil had the following parameters:

[0052] Spiral coil: 18 turns of 0.5 mm OD wire at a 1 cm pitch for an overall footprint of 27 x 27 cm, the measured inductance was 67uH; and

[0053] Square coil: 22 turns of 0.5 mm OD wire on a 30 cm square perimeter, the measured inductance was 592 uH.

[0054] The two coils were connected to the circuit, each in parallel with a 4.7 nF capacitor, giving resonant frequencies of around 250 kHz and 100 kHz in air respectively. The coils were recessed into the tabletop with a protective sheet covering them, such that as the tray passes over the top of the metal detection sensors it is scanned. In this way, the distance between the objects being scanned and the coils is minimised.

[0055] In some examples the system also includes a magnetometer, which is a passive means for detecting ferrous metallic objects (such as guns or knives). They work by detecting perturbations to a background magnetic field, for example the background magnetic field of the earth. The proximity of magnetised ferrous objects nearby adds to or subtracts from the background magnetic field, depending on their orientation. The magnetometer provides a unique signature related to the presence of ferrous metals, which can be a useful discriminator for firearms and many knives. In this example, a QST QMC5883L three-axis magneto-resistive sensor was utilised, which provides good sensing range (± 8 Gauss) and sensitivity (in the region of 2 mG of sensor noise) combined with high data output rates (up to 200 Hz). Four of these magneto-resistive sensors were connected to a subprocessing unit (e.g., a Raspberry Pi via parallel I2C interfaces). The data was read out at more than 100 frames per second, which provided data at intervals of better than 1 cm spacing in the scan direction.

[0056] In some examples the system includes both inductance coil sensor(s) and magnetometers, and in such examples the magnetometers are mounted slightly below the inductance coil(s) in the recess within the table-top and inside of the perimeter of the coil. This positions them as close as possible to the item of baggage for maximum sensitivity. The magnetometers were equally spaced across the region where the tray passes, perpendicular to the scanning direction.

[0057] Figures 3A - 3F show various alternative configurations of the system. In Figure 3A, the bulk of the configuration of the system is the same as in Figure 1. However, rather than a user manually moving the item of baggage a motor driven conveyer belt is provided. The item of baggage is moved past the sensor array by the conveyer belt. Dividers can be provided along the belt, so that items of baggage can be easily discriminated from one another. In Figure 3B, a drawer is provided into which the user places their item of baggage. The drawer is then pushed into the scanner, through a passage during which the scan is performed. A window panel is provided so that the user can see the item of baggage at all times. Once cleared, the user can walk past and pull the item of baggage out of the rear of the unit and a spring returns the drawer back out for the next user.

[0058] In Figure 3C, a belt is provided upon which the item of baggage is placed and carried past the sensor array. In contrast to the motor driven conveyer in Figure 3A, the motion here is driven by a user pushing a handle which in some examples is itself a pedestrian barrier (confining the path of the user as they walk past the system). The motion can be suitably damped so that the item of baggage travels past the sensor at a desired speed. In this configuration, a mechanical mechanism can be provided allowing the barrier handles to cycle round for each user, without impeding access to the venue.

[0059] In Figure 3D, the system has a box configuration into which the user lowers the item of baggage. Sensors in the side of the box (and, in some examples, in the base) screen the item of baggage, which can then be lifted out after a clear / alarm verdict is delivered. Figure 3E is similar to the configuration in Figure 3B, except the user places their own item of baggage on a platform through a ring of sensors and collects it from the other, open, side. Figure 3F is similar to the configuration in Figure 3B, with the addition that a barrier is provided which is mechanically linked to the drawer.

[0060] Pushing the barrier causes the bag platform to move past the sensor array. A spring or weight can be used to return the barrier and bag platform to their original position.

[0061] Figure 4A - 4C show mmW- and microwave-radar images for three example items of baggage. As the underlying physics of their interaction with an item of baggage is similar, the data they provide is discussed together. When millimetre or microwaves pass through an item of baggage, they can be reflected, scattered, absorbed, or delayed by the materials within. Some of the radar power then will be reflected from the front surface of the item of baggage (the first surface encountered, which is closest to the radar). Another portion of the radar power will be reflected from any objects within the item of baggage, and finally there will be a reflection from the surface of the tray (the surface furthest from the radar). The wide radar bandwidth of both sensors used allows for sufficient depth resolution to distinguish between the different parts of a reflected signal.

[0062] The amount of power reaching and returning from the tray surface depends on the absorption and / or scattering of the radar signal by the item of baggage and any objects within. The dielectric properties of the objects within affect the phase velocity of the radar beam, delaying the time at which the reflected beam is detected. This shifts the effective position (in depth) of the surface at which the tray surface reflection appears to occur. Similar interactions can also cause a deflection of the radar signal, or even a focussing effect.

[0063] The upper plots in Figures 4A - 4C show the region above the tray surface, and the lower plots show signals from regions penetrating below the tray. The output from the measured signal are 3D maps of radar scattered energy, with the three axes corresponding to the directionality of the radar array (e.g., the 8 elements in the millimetre radar array), the motion of the tray, and the depth-resolved sensing.

[0064] From these 3D maps, the processor (or a sub-processing unit) derives numeric values, referred to as features, which feed into the detection algorithms discussed in detail below. Some of these features rely on the signal reflected by a region of the item of baggage, and some of them on changes to the signal returned by the tray.

[0065] The principal features calculated from the 3D maps include:

[0066] Inferred depth of the item of baggage. This is the average ‘thickness’ of the item of baggage a seen by the radar, which is the distance from the first radar bin with energy in excess of a specific threshold in, and the last bin along the depth direction. This value gives an indication of the opacity of the objects within an item of baggage, with a larger inferred depth corresponding to more absorption of the radar beam.

[0067] Position, along the depth direction, of the first major scattering surface within the item of baggage.

[0068] Reflectivity of the item of baggage and objects within (and the degree of homogeneity across the item of baggage). ‘Glint’ features are also extracted, which arise from bright scattering caused by angular metallic regions such as knives inside the bag.

[0069] Centre of mass of the item of baggage (in the depth direction).

[0070] Top-down area and overall volume of radar scattering within the item of baggage. This gives an indication of how full the item of baggage is, as empty items do not return much signal.

[0071] Degree of shadowing or masking of the tray base. The fraction of the tray that is not obscured by the item of baggage.

[0072] Apparent position in the depth direction of the tray base (compared to its nominal position). This gives an indication of the dielectric content of the item of baggage. Figure 4A shows the measured signals where the item of baggage contains a steel pressure cooker; Figure 4B shows the measured signals where the item of baggage contains a large quantity of simulant and non-metallic fragmentation; and Figure 4C shows the measured signals from an item of baggage which is benign and contains items of food. The data shows a side view of the items of baggage, where the outer surface (as measured by the volume sensor) is marked with a white dashed line.

[0073] There are clear differences between the images (in both the mmW and microwave images), in the presence and location of the bright spots. These indicate where radar energy is scattered as a function of depth, and the amount of energy coming from the tray base (particularly noticeable for the mmW images). Further, the microwaves penetrate further into an item of baggage than the millimetre waves. Diffraction effects are also more pronounced for the microwave images, and so sharper edges and more distinct edges are observed for the mmW radar.

[0074] Figure 5 illustrates example data from a square 501 and spiral 502 inductance coil respectively as a metallic test object is scanned. As a metallic object moves past the coils, a shift in the resonant frequency caused by a change in inductance is observed for each coil. Object sizes and distances from the coil are typically larger than the distance between the turns of the spiral coil. This means a single peak can be expected in the response from the spiral coil, with no additional information contained within that peak. On the other hand, because the dimensions of the square coil are large compared to many of the objects within the bag, metal objects tend to cause a strong response as they move close to the first edge of the square coil, then a weaker response as they are in the middle of the coil, then another strong response as they pass the opposite edge of the coil. This leads to two peaks in signal as the tray is moved past the coil. The double peak effect is more pronounced for objects that are small as compared with the coil size, and at distances which are short as compared with the coil size.

[0075] In Figure 5, example inductance sensor data is shown from a single data run for a square 501 and spiral 502 inductance coil respectively. The response from the spiral coil has a height and width which can be measured and provided to the detection algorithm as features. The response from the square coil also has a measurable height and width, however additional information can be extracted based on the shape of the response. In this case, the ‘prominence’ of the peaks is extracted, which is a measure of how pronounced the double-peak feature is. Overall, the features depend heavily on the shape, material, and position of metallic objects within the item of baggage, and it is these factors which allow discrimination between benign and threat metallic objects.

[0076] There are occasional effects where a large non-metallic dielectric object can cause a change in impedance (via stray capacitance), and hence a change in the resonance frequency. This occurs when the object is large, and of a significant dielectric constant. For example, a 5 L of water-like material can cause a substantial ‘dip’ in the apparent measured inductance. This signature can be identified and flagged as a source for discrimination of large non-metallic lEDs.

[0077] Figures 6A and 6B show example data from a magnetometer array when scanning a simple point source magnet and a large blade respectively. Conventional use of magnetic field strength, i.e. a single measurement of the field strength at a fixed position, can be a relatively poor measure to discriminate between threat and benign objects, and so focus is placed on features of the signal that represent the size, shape, and location of ferrous metal objects within an item of baggage.

[0078] The use of magnetic closures on items of baggage, or objects within (e.g., laptops, or glasses cases) can cause issues in the use of magnetometers. The magnets typically used in these benign objects are neodymium and are therefore extremely strong and may generate a magnetic field equivalent in strength to a large firearm. However, they can be discriminated from these by observation that they are extremely small and so will appear almost as a single point source, unlikely a firearm or knife which will have a broader profile I extended distribution.

[0079] 2D maps of the magnetic field vector are produced by the magnetometers. The processor numerically attempts to find a ‘best-fit’ in the data for a model of a simple point magnetic dipole, returning the strength, position, and orientation of this best-fit dipole as features. If an item of baggage contained a magnetic fastener, this function would return ‘sensible’ values but would return ‘nonsensical’ values for objects like knives, such as implying the object at the magnetic source was positioned half a meter outside of the item of baggage volume. This is illustrated in Figures 6A and 6B, which are example data from the magnetometer array for a simple point source magnetic and large blade respectively. The top graphs show the raw magnetic field strength reading vs time for all four channels, the middle graphs show centred data with background subtraction, and the bottom graphs show the reconstructed map of magnetic field strength within the tray with the ‘x’ denoting the inferred position of the dipole. Figure 7 shows a top-down image of an example item of baggage and corresponding volume scan from a ToF sensor. In this example, the gantry height was 40 cm and so each ‘pixel’ covered a 5 cm square region. This was more than sufficient to estimate the volume of the item of baggage from a single frame with an accuracy of around 20%. At this accuracy, the system is able to distinguish small compact handbags from large bulky rucksacks. Further, the motion of the tray can be used to improve the resolution in the tray movement direction. The ToF sensor also allows the calculation of features like average bag density. The volume measurements also enabled better estimation of properties like metal content, by bounding the physical thickness of the item of baggage and therefore the relevant true possible range of distances of an object from the sensors.

[0080] Figure 8 is a plot showing how measurement of weight changes over time. One feature is extracted from the data collected by the weight sensor: the average measured weight over the duration of the scan (e.g. by measuring the weight during a time window in which the carriage is within range of the sensors, and taking a mean from the weight sensor readings within that time window). Some filtering is applied to account for the settling time when the item of baggage is dropped onto the tray, and to compensate for any additional force introduced if an operator leans on the handle too much. The vertical dashed line in Figure 8 shows the time the tray begins to move, and the dotted vertical line showing when the tray reached the end of the scanning path. The horizontal dashed line indicates the estimate of the weight of the item of baggage. The weight is also used in combination with the sensed volume to calculate an effective bag density.

[0081] In some examples, a camera module is added to the system with its field of view encompassing a portion of the scanning path. The output of this camera can be, for example, displayed to the operator allowing them to view the screened item of baggage.

[0082] The overall detection algorithm is the means by which raw sensor data (accumulated when a tray is scanned) is transformed into an actionable output (flag the item of baggage as possibly containing a threat or allow the user to collect the item and continue).

[0083] A two-stage process was implemented:

[0084] Pre-processing and feature extraction algorithms are applied, which take raw sensor data, clean it up and filter it, and identify numerical features that are expected to be characteristic of benign or threat items of baggage (as discussed above); and Classification, which uses the processed features to make benign / threat classifications on a given scan. The collection of features extracted in the first stage that can be used to describe an item of baggage is referred to as a feature vector. Features in the vector include estimated item of baggage mass, the overall outer volume of the item of baggage, the volume of the contents estimated with radar, the quantity of metallic items, the average depth of the item of baggage from the radar sensors, and the dielectric constant of any contents. There are in the region of 100 features which can be used to uniquely describe an item of baggage. In combination with a classification algorithm, it has been found that these are sufficient to distinguish between the majority of threat and benign items of baggage. Figure 9 is a schematic of data processing steps.

[0085] When the processor receives a signal that the scan is complete (e.g., one or more limit switches at the end of the scanning path have been triggered by the tray or carriage), the data collected is packaged up and passed to the feature extraction algorithm. A first stage to this is to validate that the tray moved correctly from left to right (or right to left if the system is configured to scan in reverse), and that this was all done within an appropriate speed range (neither too fast nor too slow, e.g. at least 10 cm / s and / or no more than 100 cm / s). If the operator did something unexpected, like pausing for a few seconds or reversing the scanning direction, the processing algorithm may alert the user and / or request that the scan be repeated.

[0086] Assuming that the scan was performed correctly and validly, the next stage is to map the sensor readings onto a specific point in the tray. Each piece of data from each sensor has an associated timestamp, and so with knowledge of the tray trajectory and relative position of each sensor within the system, it can be established which part of the tray each sensor was viewing at each point in time. The portion of data which is captured before the tray scan starts is also flagged and can be used to establish a background (specifically the response of each sensor when the tray is out of shot).

[0087] Data from each sensor is then processed in turn, taking the raw data captured and the processed trajectory and timing information as inputs. Each of these routines checks the data validity (e.g., for any sensor drop-outs) and subtracts off or normalises for any background measurement using the ‘tray out of shot’ portions of the data. The data is then further processed to extract out the specific features which are characteristic of aspects which define an item of baggage as suspicious or benign. The outputs from these routes are numeric ‘features’ for the feature vector. The outputs may also include processed data which can be used for diagnostics or visualisations. The results from each sensor are merged into the overall feature vector.

[0088] A further stage of feature extraction takes place once the data from all of the sensors has been processed, in which features are calculated which derive from combinations of multiple sensor data. This uses the processed sensor data and features, rather than any raw sensor data. These metafeatures include:

[0089] - Density of item of baggage, the weight divided by the optical or radar volumes;

[0090] - Spatial overlap between induction and magnetometer signals, indicative of the proportion of the metal content detected as ferrous;

[0091] - Differences in volume sensed by the optical ToF, microwave, and millimetre wave radars;

[0092] - Relative average depths probed within the item of baggage for optical ToF, microwave, and millimetre wave radar; and

[0093] - Overlap between radar and metallic centres of mass.

[0094] The final stage of the process is to use the complete feature vector for the determination of whether the item of baggage is likely to contain any threat objects or if it is benign.

[0095] In some examples, the processor uses a pre-trained machine learning based classifier. The classifier was trained on 437 unique items of baggage, with 201 examples containing threat objects and 236 containing benign objects. The items of baggage were scanned multiple times, and in different orientations, resulting in 2,889 scans corresponding to items of baggage containing benign objects and 2,482 scans corresponding to items of baggage containing threat objects. For example, each item of baggage could be scanned in six orientations in both directions of travel, resulting in a total of 12 scans per item of baggage.

[0096] Some of the items of baggage also included additional modifier objects, with the expectation that this would significantly change the feature vector of the item of baggage. The modifier objects included:

[0097] - Water bottles, canned drinks, and / or thermos flasks, with volumes of 330 mL up to 1L;

[0098] - Perishable foods: sandwiches, salads, picnics, fruit etc;

[0099] - Large clothing: coats, jackets, scarves, and jumpers;

[0100] - Large electronics: tablets, laptops, DSLR cameras; and

[0101] - E-cigarettes / vapes.

[0102] The items of baggage containing threat objects were broadly in one of four categories: (i) simulant lEDs which were principally non-metallic; (ii) simulant lEDs which were principally metallic; (iii) deactivated firearms; and (iv) bladed weapons.

[0103] The items of baggage also had varying form factors, from the following categories: (i) rucksack; (ii) satchel; (iii) handbag; or (iv) other. The distribution of the form factors within the data set was as follows:

[0104] Having established the training set, two types of machine learning classifier were considered: (i) random forest; and (ii) XGBoost (Extreme Gradient Boosting). In training the random forest model, the following steps were undertaken:

[0105] 1. Random sampling of data: A random sample of the original dataset is taken with replacement to create a subset of data (aka bootstrapping);

[0106] 2. Feature sampling: A random subset of features is chosen for each split in the decision tree. This helps to reduce the correlation between the trees and increase the diversity of individual trees.

[0107] 3. Building decision trees: A decision tree is built for each bootstrap sample using the randomly selected features. The decision tree is constructed by recursively splitting the data based on the values of the selected features until a stopping criterion is met.

[0108] 4. Aggregating the predictions: Once all of the decision trees are built, their predictions are aggregated to form the final prediction. This can be done using majority voting for classification (as was done in this case).

[0109] Figure 10 shows an ROC curve using the random forest classifier as trained on the dataset discussed above to provide a binary classification (benign or threat). Good performance is seen, with an AUC of 0.92.

[0110] In training the XGBoost model, the following steps were undertaken:

[0111] 1. Initialisation: The first step is to initialize the model with a single decision tree, which serves as the base model.

[0112] 2. Prediction and Residual Calculation: the base model is used to make predictions on the training set. The residuals (the difference between the actual values and the predicted values) are then calculated for each sample. 3. Building Trees: a new decision tree is built to predict the residuals from the previous step. This tree is added to the existing model and becomes a part of the ensemble. This process is repeated multiple times to add more trees to the ensemble.

[0113] 4. Shrinkage and Tree Pruning: to avoid overfitting, a regularization term is added to the objective function, which helps to reduce the complexity of the model. Additionally, the trees can be pruned if they do not contribute significantly to the performance of the model.

[0114] 5. Final Prediction: once all of the trees are built, the final prediction is made by combining the predictions of all of the trees in the ensemble.

[0115] The XGBoost provides several advantages: (i) it is extremely fast and can handle large datasets with high dimensionality; and (ii) it has built-in regularization, which aids in preventing overfitting and improves the generalization of the mode.

[0116] Figure 11 shows an ROC curve using an XGBoost classifier as trained on the dataset discussed above to provide a binary classification (benign or threat). It also shows good performance, with an AUC of 0.93.

[0117] With the classifiers trained previously, items of baggage were labelled binarily as either threat or benign. However multiclass classifiers were also investigated, which attempt to use the features in the data to assign a sample to one of multiple categories of classes. Using the data discussed above, a multiclass classifier was trained to label according to the following classes: (i) benign item of baggage; (ii) item of baggage containing an IED; (iii) item of baggage containing a firearm; and (iv) item of baggage containing one or more bladed weapons. Optionally the classifications can be combined into a binary determination of threat or benign at the end.

[0118] Plainly, the correct labelling of an item of baggage as threat or benign is the main priority with the type of threat item being less important. However, multiclass labelling was found to improve the classifier performance.

[0119] For example, firearms are likely to have a large metallic signature and so features that measure metallic content are important in classifying firearms. Conversely, many of the simulant lEDs are classed as non-metallic with the bulk of the content being liquid or powder, and so they will have a smaller metallic signature. A binary classifier that puts items of baggage into a single class may have conflicting goals if the feature space for the threat types is very different.

[0120] Further, a multiclass classifier can choose features to classify firearms independently of those required to classify lEDs, which leads to a more accurate overall classifier. Figure 12 shows individual ROC curves for different classes of items of baggage using a one vs rest multiclass classifier. For each ROC curve, the given class (for instance, Blades) is labelled as the positive class and the remaining classes (Benign, Firearm, and IED) are combined and labelled as the negative class. These curves illustrate the ability of the classifier to label that specific class.

[0121] In providing an overall metric for the performance of a multiclass classifier, the average of the ROC curve was taken. Here, the macro-averaging was taken where an individual ROC curve was calculated for each class and then the average of these curves taken. Figure 13 is a confusion matrix for the multiclass classifier, highlighting misclassifications. The values represent the fraction of times the predicted class was most probable. The figure shows that the main difficulty lies in distinguishing between certain types of benign items of baggage and items of baggage containing bladed weapon(s). Figure 14 shows an ROC curve for the multiclass XGBoost classifier. The multiclass AUC is approximately the same as a binary classification using this dataset, and the use of a multiclass classifier allows a further breakdown into the data.

[0122] The classifiers used above had a default set of parameters which defined how it learned from the training data, which are also referred to as hyperparameters. Work was undertaken to further tune the hyperparameters of the XGBoost multiclass classifier, and the following optimum values were found: learning_rate: 0.2; n_estimators: 150; max_depth: 3; min_child_weight: 3; subsample: 0.75; and colsample_bytree: 1.0. Figure 15 shows an ROC curve of the XGBoost classifier using the optimised hyperparameters, and displays improved performance as compared to the ROC curves in Figure 14.

[0123] Figure 16 shows features ranked by their contribution to the classification, and so indicates their relative importance. It can be seen that the weight of the item of baggage and objects contained therein is the main feature used to distinguish between benign and threat. To understand which features influence the classifier significantly, data from given sensors were removed from the training data and the classifier was retrained. Table 1 below presents the classifier performance in the form of the AUC and FPR at a TPR of 0.9 with different sensors removed, which allows the sensors to be ranked in terms of importance. The table is ordered in decreasing AUC, with a column showing the change in FPR caused by removing a given sensor. It can be seen that removing the weight sensor has the largest effect, followed by the metal detection (magnetometer and induction combined).

[0124]

[0125] Table 1

[0126] As was discussed above, a wide range of different types and mass of threat object were used to generate the training data set. Each training data sample included a metadata tag with information including the threat type, allowing the training data to be filtered prior to fitting the classifier. This allows, for example, the classifier to be trained on data sets which have no bladed weapons, or that only have threat objects with a mass greater than a threshold.

[0127] With reference to Figure 12, the multiclass classifier found bladed weapons the hardest to classify. Removing the items of baggage containing blades from the training data therefore improves the accuracy of the classifier and AUC of its ROC curve. Figure 17 shows an ROC curve for a classifier trained on IED and firearms data only (where bladed weapons were excluded). Figure 18 shows an ROC curve where the classifier was trained on a dataset where data relating to axes and generic knifes were removed. Figure 19 shows an ROC curve for a classifier which was trained on data relating to lEDs only (where firearms and bladed weapons were excluded). Figure 20 shows an ROC curve for a classifier trained on a dataset where data related to handguns, axes, and generic knives was removed. Figure 21 shows an ROC curve for threat bags weighing more than a threshold weight, and with no bladed weapons. The classifier was retrained with increasing larger minimum threat masses (nominal units used below)and the AUC and corresponding FPR at a specific TPR recorded in Table 2 below:

[0128] Table 2

[0129] Figure 22 is a plot showing the accuracy of the classifier broken down by item of baggage class. Good performance can be seen for all classifications of item of baggage, with the best performance being seen for items of baggage containing lEDs or firearms.

[0130] As an alternative to pre-processing the sensor data into features which are then used to create a riskassessment using, for example, a machine learning algorithm, a neural network may be used. The training data is used to train a type of neural network where the sensor data, which may or may not be pre-filtered or baseline corrected, are used as input layers of a neural network and the threat classes are the output layers. Once trained the neural network will directly output a risk assessment of the scanned bag without any external intermediate step of feature extraction.

[0131] As an example, the following neural network architecture could facilitate the classification of highdimensional data derived from multiple sensors to create a risk-assessment. Initially, in one example, each type of sensor data is processed through a Convolutional Neural Network (CNN), adept at handling spatial complexities and capturing distinct local patterns. Each CNN acts as a feature extractor, generating high-level feature vectors from the input sensor data. These feature vectors are subsequently concatenated, forming a unified representation of the varied sensor data. This combined vector is then fed into a Multilayer Perceptron (MLP), an interconnected system of neurons proficient at making high-level decisions based on the provided features. The MLP's output layer employs an appropriate activation function — softmax for multi-class or sigmoid for binary classification — yielding the final classification result. This architecture is thus ideal for exploiting different types of sensor data, enabling effective sensor fusion and precise data classification. While the neural network described leverages CNNs and MLP for sensor fusion and classification, other neural network-based approaches, such as autoencoders for dimensionality reduction, recurrent neural networks for sequential data, or even more complex architectures like transformer networks, can also be effectively utilized for multi-sensor data fusion classification tasks.

[0132] The systems and methods of the above embodiments may be implemented in a computer system (in particular in computer hardware or in computer software) in addition to the structural components and user interactions described.

[0133] The term “computer system” includes the hardware, software and data storage devices for embodying a system or carrying out a method according to the above-described embodiments. For example, a computer system may comprise a central processing unit (CPU), input means, output means and data storage. The computer system may have a monitor to provide a visual output display. The data storage may comprise RAM, disk drives or other computer readable media. The computer system may include a plurality of computing devices connected by a network and able to communicate with each other over that network.

[0134] The methods of the above embodiments may be provided as computer programs or as computer program products or computer readable media carrying a computer program which is arranged, when run on a computer, to perform the method(s) described above.

[0135] The term “computer readable media” includes, without limitation, any non-transitory medium or media which can be read and accessed directly by a computer or computer system. The media can include, but are not limited to, magnetic storage media such as floppy discs, hard disc storage media and magnetic tape; optical storage media such as optical discs or CD-ROMs; electrical storage media such as memory, including RAM, ROM and flash memory; and hybrids and combinations of the above such as magnetic / optical storage media.

[0136] While the disclosure has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the disclosure set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may be made without departing from the spirit and scope of the disclosure.

[0137] In particular, although the methods of the above embodiments have been described as being implemented on the systems of the embodiments described, the methods and systems of the present disclosure need not be implemented in conjunction with each other but can be implemented on alternative systems or using alternative methods respectively.

[0138] The features disclosed in the description, or in the following claims, or in the accompanying drawings, expressed in their specific forms or in terms of a means for performing the disclosed function, or a method or process for obtaining the disclosed results, as appropriate, may, separately, or in any combination of such features, be utilised for realising the disclosure in diverse forms thereof.

[0139] For the avoidance of any doubt, any theoretical explanations provided herein are provided for the purposes of improving the understanding of a reader. The inventors do not wish to be bound by any of these theoretical explanations.

[0140] Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described.

[0141] Throughout this specification, including the claims which follow, unless the context requires otherwise, the word “comprise” and “include”, and variations such as “comprises”, “comprising”, and “including” will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0142] It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” one particular value, and / or to “about” another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent “about,” it will be understood that the particular value forms another embodiment. The term “about” in relation to a numerical value is optional and means for example + / - 10%.

Claims

CLAIMS1. A system for screening items of baggage for threat items, the system comprising: a weight sensor, configured to collect data indicative of the weight of an item of baggage and objects within when the item of baggage is in contact with at least a part of the weight sensor; a sensor array, configured to collect data indicative of the item of baggage and objects within the item of baggage; and a processor, configured to receive the data indicative of the weight of the item of baggage and objects within, and the data indicative of the item of baggage and objects within the item of baggage collected by the sensor array, and derive a risk estimation from the received data.

2. The system of claim 1 , wherein the sensor array is moveable relative to the item of baggage when it is in contact with at least the part of the weight sensor.

3. The system of claims 1 or 2, wherein the system further includes a moveable baggage holder, which in use receives an item of baggage and conveys it along a scanning path, the sensor array being disposed along the scanning path.

4. The system of claim 2 or 3, further including a position sensor, configured to sense the position of the item of baggage relative to the sensor array.

5. The system of claim 3 or 4 as dependent on claim 3, the weight sensor being integrated into the moveable baggage holder.

6. The system of claim 3 or 4 as dependent on claim 3, wherein the weight sensor is directly coupled to but does not move with the moveable baggage holder.

7. The system of any of claims 3 or 4 - 6 as dependent on claim 3, wherein the moveable baggage holder being manually operable by a user.8 . The system of any of claims 3 or 4 - 6 as dependent on claim 3, the moveable baggage holder being driven by a motor so as to convey the item of baggage along the scanning path.

9. The system of any preceding claim, wherein the sensor array is provided in a housing containing a passage through which the item of baggage is conveyed.

10. The system of any preceding claim, the system further including one or more magnetometers, configured to collect data indicative of ferrous metallic items within the item of baggage and provide the data to the processor for the derivation of the risk estimation.11 . The system of any preceding claim, the system further including one or more inductive sensing coils, configured to collect data indicative of ferrous and / or non-ferrous items within the item of baggage and provide the data to the processor for the derivation of the risk estimation.

12. The system of any preceding claim, wherein the sensor array includes one or more of: a first radar sensor, configured to collect data indicative of objects within the item of baggage; a second radar sensor, configured to collect data indicative of objects within the item of baggage; and an optical depth sensor, configured to collect data indicative of the size of the item of baggage.

13. The system of any preceding claim, wherein the processor is configured to pre-process the received data before deriving the risk estimation.

14. The system of any preceding claim, wherein the processor is configured to provide the data collected by the sensor array and the data collected by the weight sensor to a pre-trained neural network, the pre-trained neural network deriving the risk estimation based on this provided data.

15. The system of any preceding claim, wherein deriving the risk estimation includes extracting one or more features from the received data to derive a feature vector describing the item of baggage.

16. The system of claim 15, wherein deriving the risk estimation includes providing the feature vector to a machine learning based classifier, the machine learning based classifier deriving the risk estimation.

17. The system of any preceding claim, wherein deriving the risk estimation includes deriving one or more meta-features from the received data.

18. The system of claim 17, wherein the one or more meta-features includes one or more of: (i) a density of the item of baggage; (ii) a spatial overlap between signals from induction and magnetometer sensors; (iii) differences in volumes sensed by each of a plurality of sensors; (iv) relative average depths probed within the item of baggage by each of a plurality of sensors; and (v)overlap between the centres of mass of the item of baggage as sensed by each of a plurality of sensors.

19. The system of any preceding claim, wherein the processor is further configured to obtain background data from the sensor array when the weight sensor indicates the presence of an item of baggage but before the item of baggage is scanned by the sensor array.

20. The system of any preceding claim, further comprising a camera, configured to obtain an image of the item of baggage.21 . A method of screening items of baggage for threat items, using the system of any preceding claim, the method comprising: weighing the item of baggage with the weight sensor; collecting data indicative of the item of baggage and objects within the item of baggage using the sensor array; and deriving the risk estimation using the processor.

Images