Methods and systems for identifying an article

Abstract

According to the present disclosure there is provided methods and systems for identifying a registered article. The method comprises obtaining a first image with a mobile device, the first image comprising an article; determining a boundary of the article in the first image; extracting a cropped image from the first image based on the boundary of the article, the cropped image comprising the article; transmitting, to a server, the cropped image; receiving, from the server, a first indication that the article matches a registered article, the first indication being based on the cropped image comprising the article; and outputting, to a user interface of the mobile device, a second indication that the article matches the registered article based on the first indication. There is also provided methods and systems for registering an article.

Description

[0001] METHODS AND SYSTEMS FOR IDENTIFYING AN ARTICLE

[0002] FIELD OF THE INVENTION

[0003] This invention relates to methods and systems for registering an article and identifying a registered article. Further, this invention relates to image processing and machine learning methods and systems. It is particularly, but not exclusively, concerned with uniquely identifying baggage and handling methods and systems, for example operating at airports, seaports, train stations, other transportation hubs or travel terminals. Additionally, the handling methods and systems discussed herein may be applied to a factory environment where an article needs to be identified from a plurality of articles.

[0004] BACKGROUND OF THE INVENTION

[0005] At present, around 2.25 billion bags are transported by the Air Transport Industry (ATI) annually. However, approximately 45,000,000 bags (2%) are mishandled. This problem is only expected to get worse, as it is predicted that 8.2 billion air passenger journeys are expected to be made in 2037. At this rate of growth, current airport processes will not be able to handle the demand and airport infrastructure and systems must be strategically planned for a sustainable future.

[0006] It is therefore very important for airlines and airports to improve baggage handling performance using scalable processes to ensure that the ATI is able to continue to provide high quality services and reduce passenger delays into the future.

[0007] The majority of mishandled bags occur where a bag tag is missing, or the bag tag was not correctly identified (for example, when the bag tag is not correctly aligned with a laser reader) and sorted incorrectly. Where a bag tag is missing, known solutions involve using the IATA standardized bag categories list and matching the identified bag category with the passenger’s description of their lost bag. Where a bag tag is not correctly aligned, a baggage handler is required to manually identify the bag using a handheld laser tag reader and to relocate the bag to the correct location. In both scenarios the rectification process is passive, requiring several labour intensive and expensive processes and devices. This is because bags that are mishandled are often very difficult to identify, and automatic tag reading (ATR) devices are expensive to implement and require constant maintenance. Further, known systems currently rely on closed circuit television (CCTV) networks for tracking purposes, which use fixed location cameras and are therefore relatively inflexible.

[0008] Moreover, even if bag tags are correctly applied, reliance on visual or manual identification of baggage using bag tags is inefficient and requires baggage handlers to manually sift through baggage to be able to access and check a bag tag. This makes prevention of bag mishandling inefficient or even impossible in certain circumstances where there are a large number of bags to review. By way of example, an aircraft cargo hold can hold several hundred suitcases in a stacked arrangement. If a baggage handler wishes to review the baggage in the cargo hold to ensure that there are no erroneously included bags, manual examination of all bags, and more importantly bag tags, is extremely time consuming or simply unfeasible. Delays caused by issues such as this can be extremely costly to airlines which may have to pay additional fees or fines as a result of late departure.

[0009] It is therefore desirable to overcome or ameliorate the above problems and limitations of existing systems and processes.

[0010] SUMMARY OF THE INVENTION

[0011] The invention is defined by the independent claims, to which reference should now be made. Preferred features are laid out in the dependent claims.

[0012] In a first aspect of the invention, there is provided a method of identifying a registered article, the method comprising: obtaining a first image with a mobile device, the first image comprising an article; determining a boundary of the article in the first image; extracting a cropped image from the first image based on the boundary of the article, the cropped image comprising the article; transmitting, to a server, the cropped image; receiving, from the server, a first indication that the article matches a registered article, the first indication being based on the cropped image comprising the article; and outputting, to a user interface of the mobile device, a second indication that the article matches the registered article based on the first indication.

[0013] In an embodiment of the invention, the boundary of the article is one of: a box comprising the article; or an outline of the article corresponding to a perimeter of the article in the first image. In an embodiment of the invention, the boundary of the article is determined using an object detection model.

[0014] In an embodiment of the invention, the object detection model determines the boundary of the article in the first image using bounding box regression.

[0015] In an embodiment of the invention, the object detection model determines the boundary of the article in the first image using region-based object detection.

[0016] An embodiment of the invention further comprises: generating a position identifier (ID), the position ID identifying a plurality of sample values in the first image corresponding to the boundary of the article; and transmitting, to the server, the position ID; receiving the first indication, the first indication comprising the position ID; and outputting the second indication based on the position ID.

[0017] An embodiment of the invention further comprises: generating a position identifier (ID), the position ID identifying a plurality of sample values in the first image corresponding to the boundary of the article; storing, in a memory of the mobile device, the position ID; and in response to receiving the first indication, retrieving the position ID from the memory; and outputting the second indication based on the position ID.

[0018] In an embodiment of the invention, the plurality of sample values are a plurality of pixels.

[0019] An embodiment of the invention further comprises: transmitting, to a server, a user identifier (ID) associated with a user of the mobile device.

[0020] In an embodiment of the invention, the second indication comprises the first image with a graphical overlay identifying the article.

[0021] In an embodiment of the invention, the graphical overlay is a box that encloses the article.

[0022] An embodiment of the invention further comprises: obtaining first metadata associated with the first image; transmitting, to the server, the first metadata; and wherein the indication is further based on the first metadata.

[0023] In an embodiment of the invention, the first metadata is one or more of: a colour of the article; an article type of the article; a geolocation of the mobile device; a departure time of a vehicle associated with the article; a target off block time of a vehicle associated with the article; a target start-up time of a vehicle associated with the article; a flight number associated with a vehicle associated with the article; a time associated with obtaining the first image; and an airport associated with the mobile device.

[0024] In an embodiment of the invention, the mobile device is one of: a mobile phone; a mobile computing device; a body-mounted camera; or a pair of smart glasses.

[0025] In an embodiment of the invention, the first indication comprises a confidence score indicative of the probability that the article matches the registered article and the second indication comprises the confidence score.

[0026] In an embodiment of the invention, the first indication comprises second metadata associated with a user associated with the article, and the second indication further comprises the second metadata.

[0027] In an embodiment of the invention, the second metadata comprises one or more of: a flight number associated with a vehicle associated with the user associated with the article; an airline code associated with a vehicle associated with the user associated with the article; an origin location of the user associated with the article; a name of the user associated with the article; a passport number associated with the user associated with the article; a baggage license plate number (LPN); a Bag Tag number; and travel information associated with the user associated with the article.

[0028] In an embodiment of the invention, the travel information includes one or more of: flight details; status updates from an article security system; updates from an article handling system; and the location of a user associated with the article relative to a destination.

[0029] In an embodiment of the invention, the article is a bag.

[0030] In an embodiment of the invention, the first image comprises a plurality of articles.

[0031] In another aspect of the invention there is provided a method of identifying a registered article, the method comprising: receiving, from a mobile device, a first image comprising an article with a plurality of characteristic features; determining a characteristic vector associated with article based on the first image, wherein the characteristic vector is defined by the plurality of characteristic features; comparing the determined characteristic vector with a plurality of registered characteristic vectors associated with a plurality of registered articles; if the determined characteristic vector matches a first registered characteristic vector of the plurality of registered characteristic vectors, the first registered characteristic vector being associated with a first registered article, generating an indication that the article matches the first registered article; and transmitting the indication to the mobile device.

[0032] In another aspect of the invention there is provided a device for identifying a registered article, the device comprising: one or more processors configured to: obtain a first image, the first image comprising an article; determine a boundary of the article in the first image; extract a cropped image from the first image based on the boundary of the article, the cropped image comprising the article; transmit, to a server, the cropped image; receive, from the server, a first indication that the article matches a registered article, the first indication being based on the cropped image comprising the article; and output, to a user interface of the device, a second indication that the article matches the registered article based on the first indication.

[0033] In another aspect of the invention there is provided a computer processing system for identifying a registered article, the computer processing system comprising: one or more processors configured to: receive, from a mobile device, a first image comprising an article with a plurality of characteristic features; determine a characteristic vector associated with article based on the first image, wherein the characteristic vector is defined by the plurality of characteristic features; compare the determined characteristic vector with a plurality of registered characteristic vectors associated with a plurality of registered articles; if the determined characteristic vector matches a first registered characteristic vector of the plurality of registered characteristic vectors, the first registered characteristic vector being associated with a first registered article, generate an indication that the article matches the first registered article; and transmit the indication to the mobile device.

[0034] In another aspect of the invention there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to: obtain a first image, the first image comprising an article; determine a boundary of the article in the first image; extract a cropped image from the first image based on the boundary of the article, the cropped image comprising the article; transmit, to a server, the cropped image; receive, from the server, a first indication that the article matches a registered article, the first indication being based on the cropped image comprising the article; and output, to a user interface, a second indication that the article matches the registered article based on the first indication.

[0035] In another aspect of the invention there is provided a method of identifying a registered article, the method comprising: obtaining a first image with a mobile device, the first image comprising a plurality of articles; determining a boundary of each article in the first image; extracting a plurality of cropped images from the first image based on the boundary of each article, each cropped image comprising an article of the plurality of articles; transmitting, to a server, each cropped image; receiving, from the server, a first indication that one of the plurality of articles matches the registered article, the first indication being based on one of the plurality of cropped images comprising the registered article; and outputting, to a user interface of the mobile device, a second indication that the one of the plurality of articles matches the registered article based on the first indication.

[0036] In another aspect of the invention there is provided a method of registering an article, the method comprising: prompting a first user of a mobile device to take a plurality of images, the plurality of images comprising an article and corresponding to a plurality of different viewing angles of the article; obtaining metadata associated with a second user of the article; associating the plurality of images with the metadata; transmitting, to a server, the plurality of images and the metadata.

[0037] In another aspect of the invention there is provided a method of registering an article, the method comprising: receiving, from a mobile device, a plurality of images, the plurality of images comprising an article with a plurality of characteristic features and corresponding to a plurality of different viewing angles of the article, and metadata associated with a user of the article; determining a characteristic vector associated with the article, wherein the characteristic vector is defined by the plurality of characteristic features; associating the metadata with the characteristic vector; storing the plurality of images, the characteristic vector and the metadata in a database.

[0038] In another aspect of the invention there is provided a mobile device for registering an article, the mobile device comprising: one or more processors configured to: prompt a first user of the mobile device to take a plurality of images, the plurality of images comprising an article and corresponding to a plurality of different viewing angles of the article; obtain metadata associated with a second user of the article; associate the plurality of images with the metadata; transmit, to a server, the plurality of images and the metadata. In another aspect of the invention there is provided a computer processing system for registering an article, the computer processing system comprising: one or more processors configured to: receive, from a mobile device, a plurality of images, the plurality of images comprising an article with a plurality of characteristic features and corresponding to a plurality of different viewing angles of the article, and metadata associated with a user of the article; determine a characteristic vector associated with the article, wherein the characteristic vector is defined by the plurality of characteristic features; associate the metadata with the characteristic vector; store the plurality of images, the characteristic vector and the metadata in a database.

[0039] In another aspect of the invention there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to: prompt a first user of the mobile device to take a plurality of images, the plurality of images comprising an article and corresponding to a plurality of different viewing angles of the article; obtain metadata associated with a second user of the article; associate the plurality of images with the metadata; transmit, to a server, the plurality of images and the metadata.

[0040] In another aspect of the invention there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to: receive, from a mobile device, a plurality of images, the plurality of images comprising an article with a plurality of characteristic features and corresponding to a plurality of different viewing angles of the article, and metadata associated with a user of the article; determine a characteristic vector associated with the article, wherein the characteristic vector is defined by the plurality of characteristic features; associate the metadata with the characteristic vector; store the plurality of images, the characteristic vector and the metadata in a database.

[0041] BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0043] Figure 1 is a schematic diagram showing an example process flow of an embodiment of the invention;

[0044] Figure 2 is a schematic diagram showing an example process for registering and identifying a registered article; Figure 3 is a schematic diagram showing an example method of comparing a query image with registered article images;

[0045] Figure 4 is an a schematic diagram showing an example method of deriving cropped images from a query image;

[0046] Figure 5 is a schematic diagram showing an example process for determining an article size / dimension;

[0047] Figure 6 is a flow diagram showing an example process flow for determining capacity for an article;

[0048] Figure 7 is an example user interface in accordance with embodiments of the invention;

[0049] Figure 8 is a schematic diagram showing an example process for comparing a characteristic vector with a plurality of registered characteristic vectors;

[0050] Figure 9A and 9B are schematic diagrams showing an example method of determining an article size / dimension and generating a 3D model;

[0051] Figure 10A and 10B are example user interfaces showing an article registration process;

[0052] Figure 11A and 11 B are example images showing enrolment and identification of a registered article;

[0053] Figure 12 is a flow diagram showing an example process flow of an embodiment of the invention;

[0054] Figure 13 is a flow diagram showing an example process flow of an embodiment of the invention;

[0055] Figure 14 is a flow diagram showing an example process flow of an embodiment of the invention;

[0056] Figure 15 is a flow diagram showing an example process flow of an embodiment of the invention.

[0057] DETAILED DESCRIPTION

[0058] The exemplary description provided herein is based on a system and method for use in the Air Transport Industry (ATI). However, it will be appreciated that the invention may find application outside this industry, including in other transportation industries or delivery industries where items are transported between locations. In particular, embodiments of the invention may be utilised in transportation terminals, on board cruise ships, security checks in buildings, parcel delivery and / or storage spaces. The following embodiments described may be implemented using a Python programming language using for example an OpenCV, TensorFlow, Keras libraries. Other suitable programming languages may also be used.

[0059] Embodiments of the invention solve the problems described above by providing a system that does not require manual scanning or examination of an article identifier to identify a particular article. Instead, object detection techniques are leveraged to identify and characterize features of an article, such as a bag, using cameras and machine learning models. In this way, the system can automatically identify an article using the characterizing features inherent to the article. This results in a significant reduction in the operational cost of identifying baggage compared to existing manual scanning or examination methods.

[0060] Instead, machine learning methods are employed in an article recognition system that is able to perform feature detection and comparison from camera inputs. This enables the system to identify and store a set of unique characteristic features (such as a dent, sticker, added marker or unusual shape) associated with an article that is used to uniquely identify the article in place of a physical article identifier, such as a traditional printed barcode bag tag. Subsequently, the article can be easily identified by performing feature detection and comparison with previously detected features. This allows baggage handlers to use camera inputs to identify baggage at a scale that is not possible with existing methods.

[0061] Accordingly, embodiments of the invention can reduce the risk of mishandling baggage by automatically establishing if the correct onward bag is loaded onto the correct departing aircraft without needing to physically scan any bags. Additionally, the system is more environmentally friendly compared to traditional systems that use paper bag tags as there is no longer a requirement for printing the bag tags on paper. The system also allows for rapid identification of baggage at any phase of the baggage handling pipeline.

[0062] Figure 1 shows a high level overview of a system 100 according to embodiments of the invention.

[0063] The system 100 comprises a mobile application implemented on one or more mobile devices 110. The mobile device 110 enable passengers or crew members to conveniently enroll their baggage, or article, 120, and any other items, before or upon arrival at an airport. This advantageously means that passenger’s articles of luggage can be enrolled anywhere using a mobile device which in turn reduces delays during a passenger journey. It will be appreciated that the term “baggage” may refer to one or more items of passenger luggage. Moreover, baggage 120 may be a bag, suitcase, or any other article to be transported. The term “article” may correspond to any item on the International Air Transport Association (IATA) standardized bag categories list.

[0064] Enrolment of an article enables those articles to be tracked more accurately and efficiently throughout the article’s journey to a final destination. For example, an airport agent may use a mobile application running on an agent device 160 to identify an unknown article 121 by interrogating a database 150 of enrolled articles.

[0065] Using the mobile device 110, a user can perform enrolment of an article 120. This process may also be referred to herein as registration of an article and can occur prior to a passenger’s arrival at an airport, or can be performed upon arrival at the airport by the passenger or airport staff. As can be appreciated from figure 1 , the mobile device 110 can enrol several different articles 120 associated with different article users. This is particularly advantageous when the articles belong to children or elderly passengers who may not be able to complete enrolment themselves. Mobile device 110 enrols an article 120 by prompting a user to take a plurality of images of the article with one or more cameras of the mobile device. The plurality of images correspond to different viewing angles of the article 120 which ensures that most or all surfaces of the article are visible in at least one of the plurality of images. This may be achieved by the mobile application configuring the cameras and / or a light detection and ranging (LiDAR) sensor of the mobile device 110 to detect the article 120 the floor, and prompting a user to capture at least one image of one or more surfaces of the article. The image may then be cropped, and in some embodiments, multiple images of each side of the article may be stitched together to form a more complete all-around perspective of the article.

[0066] The mobile device 110 also receives metadata associated with a user of the article such as a name of the user, a passport number associated with the user, travel information associated with the user, a geolocation of the mobile device, biometric data of the user, or a device identifier (ID) associated with the mobile device. Travel information can include a departure airport, destination airport, the date of the flight the user will travel on, and other such information associated with the journey the user or baggage will complete. The geolocation of the mobile device may be determined using a Global Positioning System (GPS) module incorporated into the mobile device or other positioning techniques and components. The mobile device 110 transmits the plurality of images and metadata to a server 140 which determines a characteristic vector associated with the article as further discussed herein. The mobile application interfaces with an application programming interface (API) 130 in order to enrol the one or more articles 120 in a database 150, which may be hosted on a server 140.

[0067] The camera of the mobile device 110 can be configured to provide high quality images by adjusting the shutter speed and other image capturing configurations. However, the data or image compression method may be used to improve the performance of the transfer and storage of data.

[0068] The mobile device 110 may also obtain depth information indicative of a dimension of the article 120 using one or more depth sensors of the mobile device. The depth sensors can include LiDAR or Time-of-Flight sensors. This facilitates higher accuracy measurement and detection of article features. For example, data associated with detecting the 3D details of an article surface may be used to train a more accurate matching algorithm that is better able to distinguish, for example, one red bag having a bumpy surface and another red bag having a smooth surface. The depth information obtained by the depth sensors can be transmitted to a server 140 alongside the plurality of images and metadata and also used to build a three- dimensional (3D) model of the article 120. The server 140 can advantageously extract further article features for incorporation into the characteristic vector associated with the article in addition to those identified from the plurality of images. Depth sensor information can also be used to improve the accuracy of detecting an article in an image. For example, the depth information advantageously facilitates the removal of signal noise and segmenting bags from background items shown in each image.

[0069] The system 100 may advantageously generate a passenger identifier (ID) based on the metadata associated with the user and associate the passenger ID with the article 120. This means that when the article 120 is identified at a later stage of transport, such as delivery to the cargo hold of an aircraft, the passenger ID can also be retrieved which provides comprehensive information about the article user to the baggage handler.

[0070] Once an article of luggage has been successfully enrolled, it may be deposited at a drop off location in the departure airport. Typically, the article will then travel along a predefined route for further processing, such through an article handling system, before being loaded onto an aircraft. Accordingly, system 100 may further comprise a camera array 170 for capturing further images of the passenger’s articles of luggage during this route, for example at check in or at various check points in article handing system. This advantageously ensures consistency in bag tagging and image quality. The camera array 170 therefore captures images of each article at a plurality of locations on an article journey, matches each subsequent bag image to an enrolled bag, and pairs the subsequent image with the same metadata, or “unique identifier”, associated with a user of the bag as the matched enrolled bag as described herein. This advantageously enables embodiments of the invention to track any processing anomalies or damage to the article over time by comparing historical images of the article. This is particularly useful for passengers, airlines and airports, and insurance providers for damaged or lost bags. The system may allow for estimating bag damage during the journey and excludes already damaged parts, calculating the original value of the bag, and proposing the amount of compensation if damaged happened during the journey, as well as IATA classification category in case of lost bags to help finding it. If passengers have already enrolled their bags, then if needed they can submit damaged or lost bag claims to airlines and insurance providers with a few clicks on a passenger device running the mobile application.

[0071] The system 100 may further comprise one or more external systems 180 for obtaining auxiliary information relating to the enrolled article. For example, the one or more external systems 180 may provide passenger-related information and journey-related information associated with an article of luggage. In preferred embodiments, the journey-related information may include flight details, status updates from an article security system, updates from an article handling system, and the location of the passenger relative to a destination. This information may be obtained from a number of disparate systems, such as flight records, baggage handling systems, CCTV security feeds, etc.

[0072] Mobile device 160 can separately be used to identify a registered article at various baggage handling steps. Airport ground staff can carry mobile device 160, such as a smartphone or tablet, to be able to investigate and identify baggage. For instance, when ground staff pack baggage into an aircraft cargo hold, mobile device 160 can be used to match one or more bags with enrolled bags from the enrolment phase described above. The mobile device may alternatively be a mobile phone, a mobile computing device, a body-mounted camera, or a pair of smart glasses.

[0073] Implementing embodiments of the invention on a mobile application advantageously enable the system to cover segments of an article’s journey where fixed cameras are not present, such as at the end of the Baggage Handling System (BHS) belt during loading onto a crate or manual transport to the aircraft hold.

[0074] Figure 2 shows an overview of the article registration and identification methods 200 according to embodiments of the invention.

[0075] The registration or “enrolment” phase 210 involves starting an enrolment process for checked baggage 211 b (which can correspond to article 120) and / or passenger biometric data 211a. As discussed above, a plurality of images of baggage 211 b or a user of the baggage can be obtained using camera 212 on mobile device 110 and the images sent to server 140 for enrolment. In preferred embodiments, the mobile application includes an object detection model 213 that analyses each of the plurality of images in order to identify one or more passengers 213a and one or more articles 213b shown in each image for further processing. In preferred embodiments, the object detection model 213 may be a Convolution Neural Network that includes known machine learning models such as Triplet networks and Siamese networks to determine a similarity distance score in order to determine whether an object matches a particular category or label.

[0076] In some embodiments, the object detection model 213 may be trained to identify various characteristics of passengers and / or articles using one or more specific sub-models to identify whether an identified object fits one or more predetermined categories.

[0077] For example, when identifying a bag, a type model may be trained to categorise an image of a bag according to one or more of the following predetermined categories shown below in Table 1 :

[0078] Label Name Precision N

[0079] T01 Horizontal design Hard Shell 0.000 6

[0080] T02 Upright design 0.889 476

[0081] T03 Horizontal design suitcase i-expandable 0.000 3

[0082] T05 Horizontal design suitcase andable 0.000 5

[0083] T09 Plastic / Laundry Bag 0.000 3

[0084] T10 Box 0.939 33

[0085] T12 Storage Container 0.000 5

[0086] T20 Garment Bag / Suit Carrier 0.000 5

[0087] T22 Upright design, soft material 0.000 26 T22D Upright design, combined hard and soft 0.944 748 material

[0088] T22R Upright design, hard material 0.932 2062

[0089] T25 Duffel / Sport Bag 0.379 29

[0090] T26 Lap Top / Overnight Bag 0.357 42

[0091] T27 Expandable upright 0.397 267

[0092] T28 Matted woven bag 0.000 2

[0093] T29 Backpack / Rucksack 0.083 12

[0094] Table 1 : Type Precisions of different baggage classifications determined according to an embodiment of the invention.

[0095] In addition to the types identified in Table 1 , the following additional bag categories may be defined. A label of Type 23 indicates that the bag is a horizontal design suitcase. A label of Type 6 indicates that the bag is a brief case. A label of Type 7 indicates that the bag is a document case. A label of Type 8 indicates that the bag is a military style bag. However, currently, there are no bag types indicated by the labels Type 4, Type 11 , Type 13-19, Type 21 , or Type 24.

[0096] In Table 1 , N defines the number of predictions for each bag category or name, for example “Upright design”, and the label is a standard labelling convention used in the aviation industry. Preferably, a filtering process may be used to remove very dark images based on an average brightness of pixels associated with the image.

[0097] An external elements model may be trained using the training data set of images to determine characteristics of the bag’s external elements, such as the following predetermined categories shown in Table 2:

[0098] Name Recall N_act Precision N_pred buckle 0.300 40 0.203 59 combojock 0.921 1004 0.814 1137 retractable_handle 0.943 421 0.827 480 straps Jo_close 0.650 197 0.621 206 wheel 0.988 1549 0.932 1642 zip 0.910 1539 0.914 1531 Table 2: Different external elements classifications and precisions with score threshold = 0.2. If the prediction gives a probability of less than 0.2, then the data is not included. The buckle and zip categorisations may advantageously provide for improved item classification.

[0099] Finally, a material model may be trained using the training data set of images to determine a material type of the bag. For example, the material model may categorise an image of a bag according to one or more of the following predetermined categories shown in Table 3: label name precision N

[0100] D Duel Soft / Hard 0.816 437

[0101] L Leather 0.000 3

[0102] M Metal 0.000 3

[0103] R Rigid (Hard) 0.932 1442

[0104] T Tweed 0.444 57

[0105] Table 3: Different material classifications and precisions.

[0106] It will be appreciated that similar techniques may be employed to identify and categorise a person in an image.

[0107] In some embodiments, the object detection model 213b may determine whether any of the detected articles is prohibited. In the event any article is determined to be prohibited, or fails to meet travel and airline regulations, an alert may be issued to the user of the mobile application and / or one or more security agents. Examples of prohibited articles include: explosives, such as fireworks, flares, and dynamite; flammable items, such as fuels, lighter fluid and aerosol cans; weapons, such as firearms, ammunition, knives, and other sharp objects; hazardous materials, such as chemicals, corrosives, and radioactive materials; liquids, such as containers larger than the allowable limit; sharp objects, such as box cutters, razor blades, and scissors above a certain length; tools, such as drills, saws, crowbars; banned substances and biological materials; perishable items; large batteries that exceed certain size or power limits; and self-defence items such as pepper spray, stun guns, and martial arts weapons.

[0108] Once one or more passengers and / or articles have been detected with an image, the images of the passenger and / or article may be processed in order to localise the passenger’s face 214a or the article 214b, for example by cropping the image. Further image processing techniques, such as colour balancing, may be applied at this stage.

[0109] In step 215, a plurality of characteristic features are determined for the passenger’s face and / or the article. This step may be performed at server 140 once the plurality of images or cropped images have been sent to the server. For example, the characteristic features for the article may include the article’s colour, external features or texture. The plurality of characteristic features are then used to determine a characteristic vector that uniquely describes the characteristics of the article. Each characteristic vector may be an embedding vector represented as an N-dimensional vector in a vector-space. In preferred embodiments, the embedding vectors are 128-dimensional vectors. The relative separation in the vectorspace between two embedding vectors, which each represent a different image in the database, indicates the semantic similarity between the two vectors.

[0110] In preferred embodiments, a first machine learning model determines a characteristic feature vector associated with a passenger, while a second machine learning model determines a characteristic feature vector associated with an article. These models may include known machine learning models such as Triplet networks and Siamese networks. The model is trained using a training data set of images of bags in different locations and / or from various angle viewpoint of cameras. In addition, the training data may be associated with values defining a timestamp value and an article tag numbers to uniquely identify the article of baggage.

[0111] Once one or more of the models have been trained using the training data, embodiments of the invention uses one or more trained models to identify the articles by extracting, mapping and comparing their unique features.

[0112] Each model may be trained using a convolutional neural network with a plurality of nodes. Each node has an associated weight. The neural network usually has one or more nodes forming an input layer and one or more nodes forming an output layer. Accordingly, the model may be defined by the neural network architecture with parameters defined by the weights.

[0113] Thus, it will be appreciated that the neural network is trained. However, training of neural networks is well known to the skilled person, and therefore will not be described in further detail. A unique identifier corresponding to metadata associated with a user of the article is associated with the characteristic vector and the images of the article, and the mobile application then interfaces with an API 130 and server 140 in order to provide a message containing the images, characteristic vector and unique identifier to a remote database 150.

[0114] In some embodiments, the message further comprises the current location of the first mobile device 110. This advantageously enables the application to communicate whether an enrolled item is about to be processed by one or more article handling services. For example, the location of a bag that is enrolled at a passenger’s house may be updated when further images are taken of the bag when it is deposited at a check in desk or bag drop off point. Accordingly, when identifying an unknown bag being processed at the airport by a bag handling system, the system may exclude the passenger’s enrolled bag until the location of the passenger’s bag is updated on arriving at the check in / bag drop location at the airport. This advantageously reduces the search database to enrolled articles that are known to be in the vicinity of an unknown article, and thus improves the efficiency of the search functionality.

[0115] The passenger biometrics and articles are then enrolled by storing the images, characteristic feature vectors, and unique identifier in a database in step 220. This may be achieved by the mobile application interfacing with API 130 to send a message containing the images, characteristic feature vectors, and unique identifier. In preferred embodiments, the database is a remote database 150 accessed via a server 140.

[0116] In some embodiments, auxiliary metadata such as passenger-related information is paired with the images, characteristic feature vectors and unique identifier in the database. The auxiliary information may be retrieved from one or more external data sources, such as passenger manifests, flight records and passenger name record databases. In preferred embodiments, the auxiliary information includes biometric data relating to a user associated with the article, which may be provided by the user when enrolling the article. Such auxiliary metadata can form at least part of the unique identifier described herein.

[0117] Once the images, characteristic feature vectors and metadata are stored in the database 220, the API 130 may be advantageously used to transmit an indication to enrolling device 110 that the article has been successfully registered. In a matching phase 230, an image of a scene containing baggage 231 including an unknown article 121 is captured by a camera 232 on a second mobile device 160 running the mobile application. The image may be captured by an airport agent while the unknown article 121 is travelling on a journey, for example through an article handling and sortation system. As before, the mobile application comprises an object detection model 233 that identifies and localises all bags 235 within the image in order to extract individual, cropped, images of each bag 236 identified in the image. The object detection model 233 can be used to determine a boundary of article 121 in the image, for instance using bounding box regression or regionbased object detection. Depending on the object detection model used, the boundary of the article in the image may be a box comprising the article or an outline of the article corresponding to the perimeter of the article in the image. In other words, the boundary tightly encloses the article 121 which localises an object of interest in the image, namely the article 121. Based on the determined boundary, it is possible to extract the individual, cropped, images of each bag in an image. Cropping of the image may correspond directly to the determined boundary, for instance the dimensions and location of a bounding box can be used exactly for cropping. The cropping process can be repeated for all articles detected in the image. However, the system may instead send the uncropped query image to the server. The server may then perform object detection steps using an object detection model as discussed herein to localise articles within the query image.

[0118] In step 237 the method 200 involves analysing the image of unknown article 121 in order to determine a characteristic vector that uniquely describes the characteristics of the article, as described above. The database 150 is then queried to determine whether the new characteristic vector matches any of the characteristic vectors already stored in the database. The plurality of characteristic vectors associated with registered articles are compared with the determined characteristic vector in step 237 and, if there is a match, the bag is re-identified in step 238. This can be achieved assessing the separation in the vectorspace between two characteristic vectors which is indicative of the semantic similarity between the two vectors.

[0119] If there is a match, an indication of the match can be generated and transmitted to mobile device 160. This may be achieved by using API 130 to provide a message comprising the match indication. This may referred to herein as a first indication.

[0120] In some embodiments, the system 100 may then retrieve auxiliary metadata relating to the matched article in order to determine passenger information associated with the unknown article in step 240. In preferred embodiments, the message indicating a match may include the auxiliary information relating to the matched article.

[0121] Similar or analogous considerations and methodologies apply when reidentifying a passenger in Steps 251 to 258 as for reidentifying a bag in Steps 231 to 238 described above, involving obtaining an image of a passenger, using an object detection model to detect one or more passengers in the image, localise the face and body of each identified passenger, identify one or more characteristic feature vectors associated with the identified passenger, and reidentify the passenger by comparing the characteristic feature vectors with a set of predetermined feature vectors stored in the database. In some embodiments, the geocoordinates of the passenger are paired with the image and characteristic feature vectors in the database.

[0122] In some embodiments, method 200 obtains geo-coordinates 234 associated with the bag scene image 231 , which enables the search of pre-enrolled articles in database 150 to be limited by location to create a subset of characteristic vectors, and thus be a quicker and more effective search. In some embodiments, the bag scene image 231 includes geocoordinate data, such as latitude and longitude data, but in other embodiments the system 100 may obtain the geo-coordinate data using other known techniques, such as identifying a location associated with a fixed camera that took the image, or interfacing with one or more external systems in order to determine the geo-coordinates associated with the bag image.

[0123] Figure 3 shows an example process of identifying a registered article using query image 320. When seeking to identify an unknown article, embodiments of the invention may generate a list of images that are most similar to the query image (i.e. a list of nearest neighbours). This may be achieved by searching the query database 150 for characteristic vectors that are closest, in the Euclidean distance sense, to the query image vector. This can be efficiently done, as the characteristic vectors are low-dimensional real-valued vectors. Adopting such an approach enables the system to learn to use more subtle cues, like the structure of an article’s surface or the presence of additional elements, like patterns or regions of different materials, to distinguish between similar articles.

[0124] The most similar images produce a lower distance score that can be used to identify the original article. The image may then be stored for future use cases, such as detecting whether any damage has occurred during the journey. Additionally, the image may be reproduced in information box discussed in relation to figure 7. In preferred embodiments, metadata of the passenger can be associated with the first image and used by the server to limit the characteristic vector search. Examples of metadata include a colour of the article; an article type of the article; a geolocation of the mobile device; a departure time of a vehicle associated with the article; a target off block time of a vehicle associated with the article; a target start-up time of a vehicle associated with the article; a flight number associated with a vehicle associated with the article; a time associated with obtaining the first image; and an airport associated with the mobile device. The search can also be narrowed by receiving a user identifier (ID) associated with mobile device 160. Using this user ID the server can limit the database search taking into account user ID factors such as location, role, task and so on of an airport agent operating device 160. Similarly, a user ID associated with the user of mobile device 110 can be transmitted to the server from the mobile application and used to limit the database search. In preferred embodiments, the server can take into account whether the user associated with the user ID of mobile device 110 is within a threshold distance of a departure location, such as a departure lounge or security gate. Taking this into account, the database search can be limited to only bags that can correspond to passengers at a given airport location.

[0125] As shown in the example, the machine learning model can extract a list of images 311 , 322, 313, 314 from the database of enrolled articles that have been identified as being closest matches to the query image 320. This is purely an example situation only and any number, K, of closest neighbours can be provided. When K is equal to 1 , the model only shows the most similar bag.

[0126] In Figure 3, image 311 is deemed to be most similar with a distance value of 0.2521 , while the next most similar image 312 is calculated to have a distance value of 0.5174. The least similar image 314 has a distance value of 0.7664, while the next least similar image 313 has a distance value of 0.7566. Accordingly, image 311 is considered to be the most likely to be a match for the query image 320, while image 314 is considered to be the least likely to be a match for the query image.

[0127] In some embodiments, the system determines a statistical probability that an unknown article accurately matches an enrolled article. The confidence score can be included in the message comprising the match indication. In preferred embodiments the confidence score can be used to determine if a determined characteristic vector derived from query image 320 matches a characteristic vector corresponding to a registered article by defining a threshold confidence score. For instance, a match may only be determined if the confidence score is above 90%, though it will be appreciated that other threshold scores may be used. Accordingly, the system may also produce a probability value or confidence score indicative of an identified passenger being the correct owner of an unknown article of luggage.

[0128] In some embodiments, additional steps are performed when machine learning and computer vision techniques alone are unable to uniquely identify bag. For example, the timestamp of each set of images obtained for an article of baggage may be compared against an expected journey time. A reduced set of images can be identified based on an expected time window during which the bag would pass by a particular camera location. The bag may then be uniquely identified from the subset of images. Alternatively, a shortlist of similar bags may be presented to a passenger for them to identify which bag is theirs.

[0129] The mobile application may advantageously utilize depth sensor data from the mobile device to accurately measure the size of an article of luggage, as further described below. The depth sensor data may capture surface texture details - such as whether a surface is rough or smooth - in order to further characterise a particular article of luggage. Accordingly, the use of depth sensor data not only aids in compliance with airline or aircraft specifications and regulations, and enabling real-time feedback for passengers and crew regarding onboard availability, but is also able to provide further characteristic information to uniquely identify an article.

[0130] In preferred embodiments, object detection 213 and bag measurement occur automatically and simultaneously. This may be achieved by using a machine learning model that has been trained with 3D article information, as described above, which enhances the accuracy of both object detection and object segmentation, and improves the accuracy of measurements of all details relating to the article, such as the total size of the bag, and any other additional characteristic features, such as the size of handles, dents or tears.

[0131] Figure 4 shows an example method 400 of localizing and extracting cropped images from a query image 410 to either enrol or re-identify an article. The extraction of cropped images from a query image serves multiple purposes including reducing the risk of errors in characteristic vector determination by ensuring the image comprises as few other components, such as other bags, as possible. Additionally, the use of cropped images reduces the amount of information to be transmitted. As discussed above, the mobile application on the image capturing device, such as mobile device 160, comprises an object detection model 233 that identifies and localises a bag within the image. In this example, the object detection model determines a bounding box 420 that tightly encloses the article to be identified. Using the boundary of the article in the image defined by bounding box 420, the mobile application can crop the image to obtain cropped image 330. It will also be appreciated that the boundary, which the cropping based on, does not necessarily have to be a bounding box and could conform to the outline of the article in the image more closely to form an irregular shape boundary. The corresponding cropped image may therefore have the same (or scaled) dimensions as the irregular shape.

[0132] In some embodiments, the mobile application can generate a position identifier (ID) identifying a plurality of sample values 441 , 442, 443, and 444 in the image 410. The sample values correspond to the boundary of the article by defining points in the image that correspond to at least some of the vertices of the boundary. In the example of a bounding box 420, the sample values therefore correspond to the values at corner vertices. The sample values may be individual pixels or pixel groupings in the image 410. Once the position ID is generated, it may either be stored locally in a memory of the mobile device 160 or transmitted to the server 140. If the position ID is stored locally, then the mobile device can retrieve the position ID in response to receiving an indication of a match and output an indication that the article matches a registered article based on the position ID. For instance, if the output indication is a box graphical overlay on image 410, the box graphical overlay vertices can be set equal to the sample values identified in the position ID. Accordingly, the device is able to easily provide an indication of the location of the article using the box graphical overlay without further object detection steps. This indication may referred to herein as the second indication. Two or more sample values, such as opposing corner vertices, can be used to indicate the location of the boundary of the article.

[0133] Alternatively, the position ID can be transmitted to server 140 which associates, or tags, the position ID with the determined characteristic vector derived from the query image 410. The position ID can then be included in an indication transmitted back to the mobile device 160 that there is a match. This means that the mobile device 160 can determine where to locate a box graphical overlay without further object detection steps.

[0134] In preferred embodiments, the mobile application enables automatic measurement of an article such as an item of baggage, as shown in the example 600 of Figure 6. As shown in the system 500 of Figure 5, a camera 510, such as a CCTV camera or a camera on a mobile phone, captures one or more images 511 of a bag or other item of luggage 512 belonging to a passenger. Embodiments of the invention are preferably configured to use images from mobile devices, such as from cameras of a mobile phone, tablet, laptop computer or other personal device. The bag image 511 is then processed and analysed to determine the physical real-world dimensions of the bag. In preferred embodiments, this is achieved using a bag detection and metering algorithm 521 to determine the dimensions of the bag 512.

[0135] In some embodiments, the bag detection and metering algorithm 512 establishes the dimensions of an article by determining the location of one or more corners, or vertices, of the article using data from the depth sensor. The dimensions of the article may then be determined based on the depth information.

[0136] In some embodiments, the image processing and bag detection and metering algorithm may be performed at an edge device 520. However, in embodiments that use images obtained from mobile devices, the processing and analysis of the bag image 511 may alternatively be performed on the mobile device.

[0137] Airport Operation Database (AODB) 530 is a central database or repository for all operative systems and may provide all flight-related data in real-time. For example, the AODB may provide the limitations of each aircraft arriving at a particular airport terminal such as the cabin space capacity of the particular aircraft. Accordingly, interfacing with the AODB 530 enables the system to calculate the remaining on board luggage capacity, such as remaining cabin space 531 , based on the luggage carrying capacity for a particular aircraft at the airport terminal and the total amount of luggage that has been identified for passengers checked onto a flight on the particular aircraft. It will be appreciated that although the description below relates to determining whether there is remaining cabin space for carry-on luggage, embodiments of the invention are not limited to such instances and can apply to other scenarios where there is a limited storage space for articles.

[0138] Once the dimensions of the bag 512 have been calculated by the machine learning algorithm, the bag dimensions can be paired with passenger related information, which in some embodiments may be provided from Common Use Terminal Equipment (CUTE) systems 540 once a passenger has checked in for a flight. The CUTE system 540 can provide information indicating whether a particular passenger has checked in, and integrating with CUTE system 540 enables baggage information to be associated with passenger information.

[0139] All of this data - the bag dimensions, remaining cabin size, passenger related information, and any other pertinent data - may be integrated within an Application Programming Interface (API) which enables notifications 541 to be communicated between systems. This may, for example, enable the system to notify an airport agent 550 that the amount of remaining cabin space has fallen to below a particular threshold, such as 10% or 5% of the total cabin capacity. On receiving such a notification, the airport agent can then inform subsequent passengers checking in for the flight that their carry-on bags must be placed in the hold instead of being carried into the cabin. In preferred embodiments, on determining that a carry-on bag must be placed in the hold the system is further configured to provide a bag tag for the bag. The tagged bag may then be loaded into the hold of the aircraft. In addition, the API may be implemented on a bag measurement mobile application that enables passengers to check whether their bags comply with carry-on baggage regulations at any time.

[0140] This process is described further with reference to Figure 6. However, as indicated above, it will be appreciated that embodiments of the invention are not limited to identifying the capacity of carry-on baggage, and may also be applied in other instances to automatically determine whether there is sufficient storage space to house a new item.

[0141] As shown in the example of Figure 6, in preferred embodiments, the system 600 determines whether an identified item of carry-on baggage 601 would fit into the cabin space. This may be achieved by using one or more of Time of Flight information 602a, computer vision techniques 602b and / or LiDAR information 602c in combination with machine learning 603 techniques to determine the size of an article 604. Next, a comparison algorithm 605 identifies whether there is any available room in a storage space, such as in an aircraft cabin, for a new article. This may be achieved by firstly using cameras and / or depth information to estimate the size of the article, as described above, and secondly calculating the total available space and deducting the total amount of space already occupied or reserved.

[0142] For example, in the ATI, this may be achieved by identifying the passenger or item of baggage and retrieving the flight information 606 associated with the passenger, retrieving a remaining capacity 607 for the flight, and outputting a result 608 indicating whether the bag fits or does not fit into the remaining space available. If the bag fits, the system 100 may update the remaining capacity 607. In alternative embodiments, the comparison algorithm 605 may compare the size of a bag with a maximum storage space allowable for the cabin hold. If the bag is too large, an alert may be issued.

[0143] When the maximum allowable capacity is reached for the storage space, an alert may be issued informing the relevant authority to stop accepting more articles.

[0144] Figure 7 shows example mobile device 700 with user interface 710 that can be used to identify a registered article. The interface can display an image captured by the mobile device 700 comprising one or more articles 720. Using the techniques described herein, the mobile device can localise and identify whether one or more bags in the image match a registered (or enrolled) article. As discussed herein, the server 140 can provide to mobile device 700 metadata associated with the image captured by the device once it has determined an article matches a registered article. The metadata can be information retrieved from a database such as AODB 530 and may include a flight number associated with a vehicle associated with the user associated with the article; an airline code associated with a vehicle associated with the user associated with the article; an origin location of the user associated with the article; a name of the user associated with the article; a passport number associated with the user associated with the article; a baggage license plate number (LPN); a Bag Tag number; and travel information associated with the user associated with the article. The travel information can include one or more of: flight details; status updates from an article security system; updates from an article handling system; and the location of a user associated with the article relative to a destination. The server 140 may also calculate an arrival time at which the user associated with the article will arrive at a destination based on the travel information; and if the arrival time is later than a predetermined time, transmitting the indication that article matches a registered article. This means that the indication is provided as and when it is pertinent for an airport agent to interfere and relocate baggage. The predetermined time can be one or more of a boarding opening time of a vehicle associated with the user of the article; or a boarding closing time of a vehicle associated with the user of the article.

[0145] The mobile device 700 outputs the image 700 with graphical overlay 730 to form the indication that a match has been found. The graphical overlay 730 may be a box enclosing article 720 or other appropriate graphical indications. The graphical overlay can advantageously output the metadata associated with the image and identified article 720. For example, some or all information 741 , 742, 743, 744, 746, and 747 can be output in information box 740 to provide context as to the reason for an article being in a particular stage of baggage handling. Information box 740 may be opaque or translucent. This would be particularly useful to allow ground staff to quickly identify mishandled bags at scale without needing to separately consult a register of articles to obtain information such as 741. The information can include passenger name 741 , departure location 742, destination location 743, the number of checked bags 744, a bag status (Remove Bag) 745, a match confidence score 746, enrolled images 747 captured while enrolling the bag, and a bag tag bar code.

[0146] Figure 8 shows example process 800 for determining whether an article matches a registered article. As discussed above, cropped image 810 is derived from a query image taken, for example, by an airport agent wishing to identify an article of baggage. According to the techniques discussed herein, a characteristic vector 820 is determined based on the plurality of characteristic features of the article such as colour, external features or texture. Database 850 stores the characteristic vectors 830 corresponding to registered articles which can be retrieved and compared against determined characteristic vector 820 to ascertain if there is a match. It will be appreciated that database 850 may correspond to database 150.

[0147] In preferred embodiments the determined characteristic vector 820 is compared with a subset 841 of the registered characteristic vectors 830 to drastically reduce the computational overhead required to identify a match. This is achieved as discussed herein by limiting the database search using metadata associated with the query image to select a subset of characteristic vectors that are most likely to match the determined characteristic vector. The metadata may include a colour of the article; an article type of the article; a geolocation of the mobile device; a departure time of a vehicle associated with the article; a target off block time of a vehicle associated with the article; a target start-up time of a vehicle associated with the article; a flight number associated with a vehicle associated with the article; a time associated with obtaining the first image; and an airport associated with the mobile device. Determined characteristic vector 820 can additionally or alternatively be compared with subset selected based on a user ID of mobile device 110 or mobile device 160. It is also possible to further limit the search by applying multiple constraints which results in only the registered characteristic vectors satisfying both search criteria being returned. For instance, this may be Registered Characteristic Vectors 2 and 3 if subsets 841 and 842 are both identified.

[0148] Figures 9A and 9B show example process of enrolling and obtaining dimensions of an article. First, as shown in Figure 9A, a set of example bag images 900 are obtained to improve the accuracy of the bag measurement and capturing characteristic features of the bag. As described above, multiple images 901-906 are taken of the bag in order to capture all sides of the bag and thus provide the system with images of the bag from a number of perspectives, which may be used to build a 3D model of the bag as further described below. As shown in Figure 9B, the bag detection and metering algorithm automatically identifies a bounding box 910 for a first bag image 907. This bounding box is calculated to enclose the extremities of the bag, and may be used to determine the dimensions of the article. Each vertex of bounding box 910 may be associated with depth information for that particular location, and thus the depth information and bounding box may be used to determine the overall dimensions of the bag. Each vertex of the bounding box 910 may be adjusted to better fit the enclosed bag shown. This is represented in Figure 9B by adjusted vertices 921 to 929. This process may be repeated for one or more of the other bag images 900 to identify any anomalous measurements and, in some embodiments, to determine an average set of dimensions for the article.

[0149] It will be appreciated that the above techniques may also be applied to features of an article, such as handles, dents, tears and the like. Accordingly, embodiments of the invention enable the measurement of the overall size of an article as well as the measurement of characteristic features of the article.

[0150] Figures 10A and 10B show example enrolling processes 1000 using mobile device 1010. A user can use a camera of mobile device 1010 to take images of article 1020 which can be used to register the article 1020 using the techniques described herein. The user can input user information 1030 which can be associated with the determined characteristic vector corresponding to article 1020 and can be retrieved at a later stage when the article is identified. In Figure 10B the user is prompted 1040 to take an image of article 1020 from a different viewing angle. An instruction such as “ROTATE CAMERA RIGHT” may be output to the user interface of the mobile device 1010. By providing the prompt 1040, the mobile application ensures that the characteristic features and dimensions of the article can be obtained for enrolment as described above. The mobile application may determine that one or more sides of an article have not been adequately captured, and so may prompt the user to take further images to capture any missing perspectives of the article.

[0151] Figures 11A and 11 B show example enrolment and matching images. The system may communicate with the passenger app to acquire location data, narrowing down the dataset to identify the passenger. Upon arrival at the airport, the passenger simply places their enrolled bags at a baggage self-drop system, or check-in desk, and leaves. The passenger’s enrolment can be viewed in an admin console by the airport authorities and security staff to make sure the passenger is eligible for the flight.

[0152] At any stage of the bag's journey, even if unaccompanied by the passenger or if the bag tag has been accidentally removed or lost, the agent search app can probabilistically identify the bag's owner. For example, the image shown in Figure 11 B indicates that there is a 96.7% calculated probability that the unknown article belongs to passenger “sid-test1”. An airport agent can also use the search functionality of the mobile application to determine if a particular passenger or baggage has been enrolled, and retrieve images of enrolled articles to find passengers in case bags have been left behind, lost or damaged.

[0153] By doing so, the app generates an LPN or Bag Tag number for each passenger, facilitating efficient tracking throughout the bag's journey. Additionally, the images captured serve multiple purposes, including real-time location tracking notifications for passengers, reidentification of bags across various cameras to prevent tampering, and monitoring for any changes in appearance such as damage or tampering, triggering alerts for investigation.

[0154] In a second use case, if a passenger is unable to board their flight, an agent can utilize the mobile application to swiftly identify the passenger’s bags that need removal from a stack of bags on the aircraft. Similarly, if bag tags are missing for a group of bags, the system can assist in identifying the passenger associated with each bag.

[0155] Figure 12 is a flowchart example operation 1200 of identifying a registered article. At step 1210 a mobile device obtains a first image comprising an article. At step 1220 the mobile device determines a boundary of the article in the first image. At step 1230 the mobile device extracts a cropped image from the first image based on the boundary of the article, the cropped image comprising the article. At step 1240 the mobile device transmits, to a server, the cropped image. At step 1250 the mobile device receives, from the server, a first indication that the article matches a registered article, the first indication being based on the cropped image comprising the article. At step 1260 the mobile device outputs, to a user interface of the mobile device, a second indication that the article matches the registered article based on the first indication.

[0156] Figure 13 is a flowchart operation 1300 of identifying a registered article. At step 1310 a computer processing system receives, from a mobile device, a first image comprising an article with a plurality of characteristic features. At step 1320 the computer processing system determines a characteristic vector associated with the article based on the first image, wherein the characteristic vector is defined by the plurality of characteristic features. At step 1330 the computer processing system compares the determined characteristic vector with a plurality of registered characteristic vectors associated with a plurality of registered articles. At step 1340 if the determined characteristic vector matches a first registered characteristic vector of the plurality of registered characteristic vectors, the first registered characteristic vector being associated with a first registered article, the computer processing system generates an indication that the article matches the first registered article. At step 1350 the computer processing system transmits the indication to the mobile device.

[0157] Figure 14 is a flowchart operation 1400 of registering an article. At step 1410 a mobile device prompts a first user of the mobile device to take a plurality of images, the plurality of images comprising an article and corresponding to a plurality of different viewing angles of the article. At step 1420 the mobile device obtains metadata associated with a second user of the article. At step 1430 the mobile device associates the plurality of images with the metadata. At step 1440 the mobile device transmits, to a server, the plurality of images and the metadata.

[0158] Figure 15 is a flowchart operation 1500 of registering an article. At step 1510 a computer processing system receives, from a mobile device, a plurality of images, the plurality of images comprising an article with a plurality of characteristic features and corresponding to a plurality of different viewing angles of the article, and metadata associated with a user of the article. At step 1520 the computer processing system determines a characteristic vector associated with the article, wherein the characteristic vector is defined by the plurality of characteristic features. At step 1530 the computer processing system associates the metadata with the characteristic vector. At step 1540 the computer processing system stores the plurality of images, the characteristic vector and the metadata in a database.

[0159] Additional, but non-exhaustive, use cases for the above-described system include:

[0160] 1 . Personalized Notifications: Passengers can receive personalized notifications regarding their baggage status, such as when it has been loaded onto the aircraft or is ready for collection at the destination airport.

[0161] 2. Contactless Baggage Drop-off: Enables implementation of touchless solutions for baggage drop-off, allowing passengers to scan their bags using the app and proceed without physical interaction with airport staff. 3. Lost and Found Assistance: The system can assist in locating lost bags by tracking their last known location and providing guidance to airport staff or passengers for retrieval.

[0162] 4. Baggage Damage Detection: Using image analysis, the system can detect signs of damage to baggage during handling and notify both passengers and airline staff for resolution. It can also assist airlines by stopping fraudulent claims for damaged bags if the baggage were already damaged at the time of enrolment.

[0163] 5. Baggage Re-routing: In the event of flight delays or cancellations, the system can automatically reroute passengers' baggage to their updated travel itinerary, minimizing inconvenience.

[0164] 6. Enhanced Security Screening: By analysing images of baggage contents, the system can assist security personnel in identifying potential security threats or prohibited items, enhancing overall airport security.

[0165] 7. Security Enhancement: By monitoring for any signs of tampering or unauthorized access, the system enhances airport security measures everywhere including in the baggage handling area.

[0166] 8. Efficient Baggage Handling: Airlines can use the data collected to streamline baggage handling processes, reducing delays and improving overall efficiency.

[0167] 9. Customs Compliance: The system can help ensure compliance with customs regulations by providing accurate documentation and tracking of baggage contents.

[0168] 10. Automated Baggage Claim: The system can streamline the baggage claim process by accurately identifying and sorting bags upon arrival, reducing waiting times for passengers.

[0169] 11 . Optimized Baggage Loading: Airlines can use the data collected to optimize the loading of baggage onto aircraft, ensuring even distribution of weight and minimizing fuel consumption.

[0170] 12. Lost Baggage Prevention: The system can proactively identify bags at risk of being misplaced or lost during transit and alert airport staff for immediate intervention.

[0171] 13. Baggage Tracking Beyond Airports: Extend baggage tracking capabilities to other modes of transportation, such as trains or buses, for a seamless end-to- end travel experience and hotels.

[0172] 14. Data Analytics for Performance Improvement: Airlines and airports can leverage the data collected by the system for analytics purposes, identifying trends, optimizing processes, and improving overall performance. 15. Premium Baggage Services: Offer premium services such as priority handling or baggage concierge services based on passengers' preferences and travel history that can be selected within the app.

[0173] 16. Integration with Loyalty Programs: Integrate baggage tracking and handling preferences with loyalty programs, offering enhanced benefits to frequent flyers.

[0174] The above described system 100 may interact with other airport systems in order to output the determined bag type or / and colour to other systems. This may be performed by way of Web Services Description Language, WSDL, Simple Article Access Protocol (SOAP), or Extensible Markup Language, XML, or using a REST\JSON API call but other messaging protocols for exchanging structured information over a network will be known to the skilled person.

[0175] From the foregoing, it will be appreciated that the system, device and method may include a computing device, such as a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a mobile telephone, a smartphone. This may be advantageously used to capture an image of a bag at any location and may be communicatively coupled to a cloud web service hosting the algorithm.

[0176] The device may comprise a computer processor running one or more server processes for communicating with client devices. The server processes comprise computer readable program instructions for carrying out the operations of the present invention. The computer readable program instructions may be or source code or article code written in or in any combination of suitable programming languages including procedural programming languages such as Python, C, article orientated programming languages such as C#, C++, Java, object orientated programming languages such as Swift or Kotlin, and their related libraries and modules.

[0177] Exemplary embodiments of the invention may be implemented as a circuit board which may include a CPU, a bus, RAM, flash memory, one or more ports for operation of connected I / O apparatus such as printers, display, keypads, sensors and cameras, ROM, and the like.

[0178] The wired or wireless communication networks described above may be public, private, wired or wireless network. The communications network may include one or more of a local area network (LAN), a wide area network (WAN), the Internet, a mobile telephony communication system, or a satellite communication system. The communications network may comprise any suitable infrastructure, including copper cables, optical cables or fibres, routers, firewalls, switches, gateway computers and edge servers.

[0179] The system described above may comprise a Graphical User Interface. Embodiments of the invention may include an on-screen graphical user interface. The user interface may be provided, for example, in the form of a widget embedded in a web site, as an application for a device, or on a dedicated landing web page. Computer readable program instructions for implementing the graphical user interface may be downloaded to the client device from a computer readable storage medium via a network, for example, the Internet, a local area network (LAN), a wide area network (WAN) and / or a wireless network. The instructions may be stored in a computer readable storage medium within the client device.

[0180] As will be appreciated by one of skill in the art, the invention described herein may be embodied in whole or in part as a method, a data processing system, or a computer program product including computer readable instructions. Accordingly, the invention may take the form of an entirely hardware embodiment or an embodiment combining software, hardware and any other suitable approach or apparatus.

[0181] The computer readable program instructions may be stored on a non-transitory, tangible computer readable medium. The computer readable storage medium may include one or more of an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk.

[0182] The above detailed description of embodiments of the invention is not intended to be exhaustive or to limit the invention to the precise form disclosed. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel or may be performed at different times. The teachings of the invention provided herein can be applied to other systems, not necessarily the system described above. The elements and acts of the various embodiments described above can be combined to provide further embodiments. While some embodiments of the inventions have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the disclosure. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the disclosure.

Claims

CLAIMS1 . A method of identifying a registered article, the method comprising: obtaining a first image with a mobile device, the first image comprising an article; determining a boundary of the article in the first image; extracting a cropped image from the first image based on the boundary of the article, the cropped image comprising the article; transmitting, to a server, the cropped image; receiving, from the server, a first indication that the article matches a registered article, the first indication being based on the cropped image comprising the article; and outputting, to a user interface of the mobile device, a second indication that the article matches the registered article based on the first indication.

2. The method of claim 1 , wherein the boundary of the article is one of: a box comprising the article; or an outline of the article corresponding to a perimeter of the article in the first image.

3. The method of claim 1 , wherein the boundary of the article is determined using an object detection model.

4. The method of claim 3, wherein the object detection model determines the boundary of the article in the first image using bounding box regression.

5. The method of claim 3, wherein the object detection model determines the boundary of the article in the first image using region-based object detection.

6. The method of any preceding claim, comprising: generating a position identifier (ID), the position ID identifying a plurality of sample values in the first image corresponding to the boundary of the article; and transmitting, to the server, the position ID; receiving the first indication, the first indication comprising the position ID; and outputting the second indication based on the position ID.

7. The method of any of claims 1 to 5, comprising:generating a position identifier (ID), the position ID identifying a plurality of sample values in the first image corresponding to the boundary of the article; storing, in a memory of the mobile device, the position ID; and in response to receiving the first indication, retrieving the position ID from the memory; and outputting the second indication based on the position ID.

8. The method of claim 6 or 7, wherein the plurality of sample values are a plurality of pixels.

9. The method of any preceding claim, comprising: transmitting, to a server, a user identifier (ID) associated with a user of the mobile device.

10. The method of any preceding claim, wherein the second indication comprises the first image with a graphical overlay identifying the article.11 . The method of claim 10, wherein the graphical overlay is a box that encloses the article.

12. The method of any preceding claim, comprising: obtaining first metadata associated with the first image; transmitting, to the server, the first metadata; and wherein the first indication is further based on the first metadata.

13. The method of claim 12, wherein the first metadata is one or more of: a colour of the article; an article type of the article; a geolocation of the mobile device; a departure time of a vehicle associated with the article; a target off block time of a vehicle associated with the article; a target start-up time of a vehicle associated with the article; a flight number associated with a vehicle associated with the article; a time associated with obtaining the first image; and an airport associated with the mobile device.

14. The method of any preceding claim, wherein the mobile device is one of: a mobile phone; a mobile computing device; a body-mounted camera; or a pair of smart glasses.

15. The method of any preceding claim, wherein the first indication comprises a confidence score indicative of the probability that the article matches the registered article and the second indication comprises the confidence score.

16. The method of any preceding claim, wherein the first indication comprises second metadata associated with a user associated with the article, and the second indication further comprises the second metadata.

17. The method of claim 16, wherein the second metadata comprises one or more of: a flight number associated with a vehicle associated with the user associated with the article; an airline code associated with a vehicle associated with the user associated with the article; an origin location of the user associated with the article; a name of the user associated with the article; a passport number associated with the user associated with the article; a baggage license plate number (LPN); a Bag Tag number; and travel information associated with the user associated with the article.

18. The method of claim 17, wherein the travel information includes one or more of: flight details; status updates from an article security system; updates from an article handling system; and the location of a user associated with the article relative to a destination.

19. The method of any preceding claim, wherein the article is a bag.

20. The method of any preceding claim, wherein the first image comprises a plurality of articles.

21. A method of identifying a registered article, the method comprising: receiving, from a mobile device, a first image comprising an article with a plurality of characteristic features; determining a characteristic vector associated with article based on the first image, wherein the characteristic vector is defined by the plurality of characteristic features; comparing the determined characteristic vector with a plurality of registered characteristic vectors associated with a plurality of registered articles; if the determined characteristic vector matches a first registered characteristic vector of the plurality of registered characteristic vectors, the first registered characteristic vector being associated with a first registered article, generating an indication that the article matches the first registered article; and transmitting the indication to the mobile device.

22. The method of claim 21 , comprising: receiving an position identifier (ID) identifying a plurality of sample values in a second image obtained with the mobile device, the first image having been cropped from the second image by extracting the first image based on a determined boundary of the article; associating the position ID with the determined characteristic vector; including the position ID in the generated indication based on the determined characteristic vector matching the first registered characteristic vector.

23. The method of claim 22, wherein the plurality of sample values are a plurality of pixels.

24. The method of claim 21 , comprising: receiving first metadata associated with the second image; wherein the determined characteristic vector is compared with a subset of the plurality of registered characteristic vectors, the subset having been selected based on the first metadata.

25. The method of claim 24, wherein the first metadata is one or more of: a colour of the article; an article type of the article; a geolocation of the mobile device;a departure time of a vehicle associated with the article; a target off block time of a vehicle associated with the article; a target start-up time of a vehicle associated with the article; a flight number associated with a vehicle associated with the article; a time associated with obtaining the first image; and an airport associated with the mobile device.

26. The method of any of claims 21 to 25, comprising: receiving a first user identifier (ID) associated with a user of the mobile device; wherein the determined characteristic vector is compared with a subset of the plurality of registered characteristic vectors, the subset having been selected based on the first user ID.

27. The method of any of claims 21 to 26, comprising: receiving a second user identifier (ID) associated with a user of a second mobile device, the second mobile device being within a threshold distance of a departure location of the article; wherein the characteristic vector is compared with a subset of the plurality of registered characteristic vectors, the subset having been selected based on the second user ID.

28. The method of claim 27, comprising: associating the second user ID with the determined characteristic vector; including the second user ID in the generated indication based on the determined characteristic vector matching the first registered characteristic vector.

29. The method of any of claims 21 to 28, comprising: determining a confidence score indicating the probability that the determined characteristic vector matches the first registered characteristic vector; wherein the determined characteristic vector matches the first registered characteristic vector if the confidence score is above a threshold score.

30. The method of claim 29, wherein the indication comprises the confidence score.31 . The method of any of claims 21 to 30, wherein the indication comprises second metadata associated with user associated with the article.

32. The method of claim 31 , wherein the second metadata comprises one or more of: a flight number associated with a vehicle associated with the user associated with the article; an airline code associated with a vehicle associated with the user associated with the article; an origin location of the user associated with the article; a name of the user associated with the article; a passport number associated with the user associated with the article; a baggage license plate number (LPN); a Bag Tag number; and travel information associated with the user associated with the article.

33. The method of claim 32, wherein the travel information includes one or more of: flight details; status updates from an article security system; updates from an article handling system; and the location of the user associated with the article relative to a destination.

34. The method of claim 32 or 33, comprising: calculating an arrival time at which the user associated with the article will arrive at a destination based on the travel information; and if the arrival time is later than a predetermined time, transmitting the indication.

35. The method of claim 34, wherein the predetermined time is one of: a boarding opening time of a vehicle associated with the user of the article; or a boarding closing time of a vehicle associated with the user of the article.

36. The method of any of claims 21 to 35, wherein the article is a bag.

37. A device for identifying a registered article, the device comprising: one or more processors configured to: obtain a first image, the first image comprising an article; determine a boundary of the article in the first image;extract a cropped image from the first image based on the boundary of the article, the cropped image comprising the article; transmit, to a server, the cropped image; receive, from the server, a first indication that the article matches a registered article, the first indication being based on the cropped image comprising the article; and output, to a user interface of the device, a second indication that the article matches the registered article based on the first indication.

38. A computer processing system for identifying a registered article, the computer processing system comprising: one or more processors configured to: receive, from a mobile device, a first image comprising an article with a plurality of characteristic features; determine a characteristic vector associated with article based on the first image, wherein the characteristic vector is defined by the plurality of characteristic features; compare the determined characteristic vector with a plurality of registered characteristic vectors associated with a plurality of registered articles; if the determined characteristic vector matches a first registered characteristic vector of the plurality of registered characteristic vectors, the first registered characteristic vector being associated with a first registered article, generate an indication that the article matches the first registered article; and transmit the indication to the mobile device.

39. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to: obtain a first image, the first image comprising an article; determine a boundary of the article in the first image; extract a cropped image from the first image based on the boundary of the article, the cropped image comprising the article; transmit, to a server, the cropped image;receive, from the server, a first indication that the article matches a registered article, the first indication being based on the cropped image comprising the article; and output, to a user interface, a second indication that the article matches the registered article based on the first indication.

40. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to: receive, from a mobile device, a first image comprising an article with a plurality of characteristic features; determine a characteristic vector associated with article based on the first image, wherein the characteristic vector is defined by the plurality of characteristic features; compare the determined characteristic vector with a plurality of registered characteristic vectors associated with a plurality of registered articles; if the determined characteristic vector matches a first registered characteristic vector of the plurality of registered characteristic vectors, the first registered characteristic vector being associated with a first registered article, generate an indication that the article matches the first registered article; and transmit the indication to the mobile device.

41. A method of identifying a registered article, the method comprising: obtaining a first image with a mobile device, the first image comprising a plurality of articles; determining a boundary of each article in the first image; extracting a plurality of cropped images from the first image based on the boundary of each article, each cropped image comprising an article of the plurality of articles; transmitting, to a server, each cropped image; receiving, from the server, a first indication that one of the plurality of articles matches the registered article, the first indication being based on one of the plurality of cropped images comprising the registered article; and outputting, to a user interface of the mobile device, a second indication that the one of the plurality of articles matches the registered article based on the first indication.

42. The method of claim 41 wherein the second indication comprises the first image with a graphical overlay identifying the article and preferably wherein the graphicaloverlay corresponds to the position, within the first image, of the cropped image associated with the matching article.

43. A method of registering an article, the method comprising: prompting a first user of a mobile device to take a plurality of images, the plurality of images comprising an article and corresponding to a plurality of different viewing angles of the article; obtaining metadata associated with a second user of the article; associating the plurality of images with the metadata; transmitting, to a server, the plurality of images and the metadata.

44. The method of claim 43, wherein the metadata includes one or more of: a name of the second user; a passport number associated with the second user; travel information associated with the second user; a geolocation of the mobile device; biometric data of the second user; or a device identifier (ID) associated with the mobile device.

45. The method of claim 43 or 44, comprising: obtaining, with a depth sensor of the mobile device, depth information indicative of a dimension of the article; and transmitting, to the server, the depth information.

46. The method of any of claims 43 to 45, comprising: receiving, from the server, an indication that the article has been registered.

47. The method of any of claims 43 to 46, wherein the first user is the second user.

48. The method of any of claims 43 to 47, wherein the article is a bag.

49. The method of any of claims 43 to 48, comprising: determining whether the article is a prohibited article with an object detection model, and issuing an alert if the first article is a prohibited article.

50. A method of registering an article, the method comprising: receiving, from a mobile device, a plurality of images, the plurality of images comprising an article with a plurality of characteristic features and corresponding to a plurality of different viewing angles of the article, and metadata associated with a user of the article; determining a characteristic vector associated with the article, wherein the characteristic vector is defined by the plurality of characteristic features; associating the metadata with the characteristic vector; storing the plurality of images, the characteristic vector and the metadata in a database.51 . The method of claim 48, wherein the metadata includes one or more of: a name of the second user; a passport number associated with the second user; travel information associated with the second user; a geolocation of the mobile device; biometric data of the second user; or a device identifier (ID) associated with the mobile device.

52. The method of claim 48 or 49, comprising: receiving depth information indicative of a dimension the article; and wherein the characteristic vector is further defined by the depth information.

53. The method of any of claims 48 to 50, comprising; transmitting, to the mobile device, an indication that the article has been registered in response to storing the plurality of images, the characteristic vector and the metadata in the database.

54. The method of any of claims 49 to 52, wherein the article is a bag.

55. A mobile device for registering an article, the mobile device comprising: one or more processors configured to: prompt a first user of the mobile device to take a plurality of images, the plurality of images comprising an article and corresponding to a plurality of different viewing angles of the article; obtain metadata associated with a second user of the article;associate the plurality of images with the metadata; transmit, to a server, the plurality of images and the metadata.

56. A computer processing system for registering an article, the computer processing system comprising: one or more processors configured to: receive, from a mobile device, a plurality of images, the plurality of images comprising an article with a plurality of characteristic features and corresponding to a plurality of different viewing angles of the article, and metadata associated with a user of the article; determine a characteristic vector associated with the article, wherein the characteristic vector is defined by the plurality of characteristic features; associate the metadata with the characteristic vector; store the plurality of images, the characteristic vector and the metadata in a database.

57. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to: prompt a first user of the mobile device to take a plurality of images, the plurality of images comprising an article and corresponding to a plurality of different viewing angles of the article; obtain metadata associated with a second user of the article; associate the plurality of images with the metadata; transmit, to a server, the plurality of images and the metadata.

58. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to: receive, from a mobile device, a plurality of images, the plurality of images comprising an article with a plurality of characteristic features and corresponding to a plurality of different viewing angles of the article, and metadata associated with a user of the article; determine a characteristic vector associated with the article, wherein the characteristic vector is defined by the plurality of characteristic features; associate the metadata with the characteristic vector; store the plurality of images, the characteristic vector and the metadata in a database.

Images