System and method for linking checked baggage of airport bus using ai-based recognition and GPS coordinates

Abstract

Embodiments relate to a system and a method for linking checked baggage of an airport bus using AI-based recognition and GPS coordinates, the system comprising a terminal device of an airport bus that: calculates GPS data of the airport bus; generates face image data of a user by photographing a face of the user boarding the airport bus; calculates identity data of the user by analyzing the face image of the user; authenticates an intention of the user to board an aircraft on the basis of GPS data of the user received from a terminal device of the user whose identity has been confirmed and the GPS data of the airport bus; requests and obtains mobile aircraft boarding pass data of the authenticated user from the terminal device of the user; generates a baggage linkage code corresponding to the mobile boarding pass of the user on the basis of article data of the user boarding the airport bus; and transmits baggage linkage data including the baggage linkage code to the terminal device of the user via a server.

Description

System and method for linking checked baggage on airport buses using AI-based recognition and GPS coordinates

[0001] Embodiments of the present application relate to a system and method for linking checked baggage loaded onto an airport bus by a passenger at the time of boarding the airport bus to an airport security screening system within the airport by utilizing an aircraft boarding intention authentication technology utilizing AI-based recognition and GPS coordinate-based location identification technology.

[0002] When departing by air, passengers must either carry their baggage on board as carry-on baggage or check it in the aircraft overhead bin. Carry-on baggage regulations vary by airline, but the weight allowance for carry-on baggage generally requires that the combined length, width, and height of the baggage compartment be less than 20 inches, while also meeting weight limits. Therefore, passengers with a large amount of baggage must check their baggage.

[0003] The checked baggage handling process begins with passengers arriving at the airport, checking in at the airline-designated check-in counter within the airport to purchase their boarding pass or confirm their reservation, and then checking in their baggage. This checked baggage handling process can be quite lengthy when there are a lot of people waiting at the check-in counter, which is a major inconvenience for passengers.

[0004] With the advancement of mobile technology, advance check-in (or mobile check-in) is becoming more popular, allowing passengers to pre-fill their boarding pass before arriving at the airport without having to obtain a boarding pass at the check-in counter.

[0005] However, even passengers who have checked in in advance must visit the check-in counter to check in their luggage, so there is a problem in that they cannot be free from waiting at the check-in counter.

[0006] To ensure that the effectiveness of advance check-in is not diminished, technology is required that eliminates the need to visit designated check-in points within the airport, such as check-in counters.

[0007] Based on the discussion described above, embodiments of the present application provide a system and method for linking checked baggage of an airport bus using AI-based recognition and GPS coordinates, which is configured to link checked baggage loaded on an airport bus by a passenger at the time of boarding the airport bus to an airport security screening system within the airport by utilizing an aircraft boarding intention authentication technology using AI-based recognition and GPS coordinate-based location identification technology.

[0008] A method for linking checked baggage of an airport bus using AI-based recognition and GPS coordinates according to one aspect of the present application comprises the steps of: obtaining, by a terminal device of a user who will board an aircraft with baggage, mobile airport bus boarding pass data for the airport bus that the user will use to arrive at the airport to board the aircraft; analyzing, by the terminal device of the user, a baggage bag image photographed of the user's baggage to be loaded onto the aircraft and an image of the user's baggage showing the inside of the user's baggage, thereby calculating the user's baggage data, and transmitting the user's baggage data to a server; analyzing, by the terminal device of the airport bus, a facial image of the user's face photographed of the user boarding the airport bus, thereby verifying the user's identity; authenticating, by the terminal device of the airport bus, the user's intention to board the aircraft based on the GPS data of the user received from the terminal device of the user whose identity has been verified and the GPS data of the airport bus; A step of requesting and obtaining mobile boarding pass data for an aircraft of a user whose intention to board the aircraft has been authenticated by a terminal device of the airport bus, and generating a baggage linkage code corresponding to the mobile boarding pass of the user based on the item data of the user who boarded the airport bus, and generating the baggage linkage code; - The user's terminal device pre-storing the user's mobile boarding pass data for the aircraft through a pre-check-in process; A step of transmitting the user's identity verification result and the baggage linkage code to a server by the terminal device of the airport bus; A step of transmitting the user's baggage linkage code to the user's terminal device based on the user's identity verification result, from the server;In the above server, the step of receiving a baggage bag image captured by an airport security screening system of an airport and reading result data for an X-RAY image of the baggage bag, wherein the reading result includes an identification result for at least one item contained in the baggage bag of the user, a step of associating the reading result data for the baggage bag with a baggage linkage code of the user based on a matching result of the baggage bag image of the airport security screening system and the baggage bag image of the user, and a matching result of the reading result data of the airport security screening system and the item data of the user; a step of transmitting a notification message to a terminal device of the user based on a message regarding the baggage screening result of the user received from the airport security screening system;

[0009] In one embodiment, the step of authenticating the user's intention to board an aircraft based on the user's GPS data and the airport bus's GPS data received from the user's terminal whose identity has been verified may include the step of determining whether the airport bus has entered a preset designated section based on the GPS data of the airport bus, wherein the preset section is a certain section including an airport regular route among the entire operation sections of the airport bus; the step of determining whether the user's GPS location matches the airport bus's GPS location at at least one point on the preset designated section if the airport bus has entered the preset designated section; and the step of acknowledging the user's intention to board an aircraft if the user's GPS location and the airport bus' GPS location are determined to match each other at at least one point on the preset designated section.

[0010] In one embodiment, the step of analyzing the user's luggage image showing the inside of the user's luggage bag and calculating the user's item data may include the steps of: applying the user's luggage image to a first deep learning-based recognition model to recognize the user's luggage bag; retrieving weight data of the user's luggage bag through the Internet or a server based on a recognition result of the luggage bag; applying the user's luggage image to a deep learning-based segmentation model to segment an item area in which luggage appears in the user's luggage image by class of luggage and extracting the segmented item area as a sub-image; applying the sub-image of the extracted item area to a second deep learning-based recognition model to recognize an item appearing in the user's luggage image; retrieving weight data of an item in the user's luggage bag through the Internet or a server based on a recognition result of the item; and calculating weight information of the user's luggage based on a search result of the weight of the recognized luggage bag and a search result of the weight of the recognized item.

[0011] In one embodiment, the step of calculating the user's item data may further include a step of requesting input of information about items contained within a pouch if the class of the recognized item corresponds to a pouch. Furthermore, in some embodiments, the step of calculating may further include a step of retrieving weight data for the pouch based on the input information about the items within the pouch and size data of the pouch calculated from the segmentation result of the item area in which the pouch appears.

[0012] In one embodiment, the deep learning-based segmentation model is trained using a first training data set, the first training data set is comprised of a plurality of training samples, each training sample comprising training data and label data. The training data includes at least one training image, a sample image of an incomplete shape representing a learning sample item having an incomplete shape, and a sample image of a complete shape representing an item having a complete shape as an item shown in the sample image of the incomplete shape. The label data represents an actual value of an item type of luggage shown in the training image, and the luggage image is input to the trained deep learning-based segmentation model.

[0013] In one embodiment, the second deep learning-based recognition model may include an input layer that inputs the sub-image as an input image, an encoder including one or more encoding blocks, a decoder including one or more decoding blocks, and a classification layer that recognizes and outputs an item appearing in the sub-image based on an encoding vector output from the decoder.

[0014] The second deep learning-based recognition model may be configured to divide an input sub-image into a plurality of patches, each patch having a fixed size, convert each patch into a patch embedding vector, combine a position embedding value indicating the position of each patch with the patch embedding vector of the corresponding patch, input the combined vector into the encoder for processing to produce an encoding vector, recognize an item appearing in the sub-image based on the encoding vector, and output a recognition result.

[0015] The above encoder may output class tokens as the encoding vector through a multi-head attention mechanism, and may have learned relationships between multiple patches through the multi-head attention mechanism.

[0016] In one embodiment, the second deep learning-based recognition model may be a foundation model pre-trained with a specific data set that is fine-tuned to recognize imperfectly shaped or fully shaped items contained in a luggage bag.

[0017] In one embodiment, the step of calculating the user's item data may further include the step of calculating size information of an item appearing in an item area based on the size information of the luggage bag and the segmentation result. The step of recognizing an item appearing in the user's luggage image applies the corresponding sub-image in which the luggage appears and the size information of the item to the second deep learning-based recognition model.

[0018] The second deep learning-based recognition model may convert the size information of the normalized item into a size information tensor, generate tensor data of the baggage based on the feature map and size information tensor extracted from the sub-image, input the generated tensor data of the baggage into a fully connected layer as an intermediate operation result, and through the fully connected layer, the second deep learning-based recognition model may recognize an item appearing in a sub-image having an incomplete shape by considering the characteristics of the baggage having an incomplete shape.

[0019] In one embodiment, the step of transmitting the notification message may include: when receiving a drop message indicating that checked baggage has been searched in baggage screening from the airport security screening system, transmitting a first notification message to the user's terminal device based on a baggage linkage code in the drop message; wherein the drop message includes the baggage linkage code, and the first notification message includes a baggage check location and the baggage linkage code; and when receiving a pass message indicating that checked baggage has not been searched in baggage screening from the airport security screening system, transmitting a second notification message including a baggage linkage code to the user's terminal device based on the baggage linkage code in the pass message from the airport security screening system (400).

[0020] According to another aspect of the present application, a computer-readable recording medium can record a program for performing the checked baggage linking method of an airport bus according to the above-described embodiments.

[0021] According to another aspect of the present application, a system for linking checked baggage of an airport bus using AI-based recognition and GPS coordinates comprises: a user's terminal device that acquires and stores mobile boarding pass data for an airport bus by performing a purchase process for a boarding pass for an airport bus that a user will use to arrive at an airport to board an airplane; generates a baggage image by photographing the user's baggage to be loaded onto the airplane and a baggage image by photographing the inside of the user's baggage; analyzes the user's baggage image to calculate the user's item data; transmits the baggage image by photographing the user's baggage to be loaded onto the airplane and the user's baggage image and the user's baggage data to a server; performs a pre-check-in process for the airplane to obtain and store mobile boarding pass data for the airplane; calculates GPS data of an airport bus; generates facial image data of the user by photographing the face of the user boarding the airport bus; analyzes the user's facial image to calculate identity data of the user; authenticates the user's intention to board an airplane based on the GPS data of the user received from the terminal device of the identified user and the GPS data of the airport bus; and requests the authenticated user's mobile airplane boarding pass data to the terminal device of the user. A terminal device of an airport bus that acquires and generates a baggage linkage code corresponding to a mobile boarding pass of a user based on the item data of the user who boarded the airport bus, and transmits the baggage linkage data including the baggage linkage code to the terminal device of the user through a server, and receives a baggage bag image taken of the baggage bag from an airport security inspection system installed at an airport and reading result data for an X-RAY image of the baggage bag, wherein the reading result includes an identification result for at least one item contained in the baggage bag of the user.A server may be included that receives an image of the user's baggage and data of the user's goods from the user's terminal device, associates the reading result data for the baggage with the user's baggage linkage code based on a matching result of the baggage bag image of the airport security screening system and the image of the user's baggage and a matching result of the reading result data of the airport security screening system and the data of the user's goods, and transmits a notification message to the user's terminal device based on a message regarding the user's baggage screening result received from the airport security screening system.

[0022] The system and method for linking checked baggage on an airport bus using AI-based recognition and GPS coordinates according to various embodiments of the present application can link checked baggage loaded onto an airport bus by a passenger to the airport security screening system within the airport, thereby eliminating the burden of passengers scheduled to board an aircraft having to visit a designated location within the airport, such as a check-in counter. As a result, passengers can minimize the burden of time required to complete check-in and check-in procedures prior to boarding. Ultimately, passengers can use the aircraft quickly and conveniently.

[0023] Additionally, the effect of skipping the check-in desk visit due to this checked baggage linkage can provide additional benefits of energy savings and securing the transportation competitiveness of airport buses, as it increases the likelihood that users who plan to board an aircraft will use airport buses instead of personal vehicles when approaching the airport.

[0024] In addition, the effect of omitting a visit to the check-in desk due to this checked baggage linkage allows the existing check-in desk space, including the passenger waiting area, to be replaced with other internal spaces, ultimately allowing for more efficient use of internal space for boarding aircraft users.

[0025] Figure 1 illustrates a network environment of a system for linking checked baggage of an airport bus using AI-based recognition and GPS coordinates according to one aspect of the present application.

[0026] FIG. 2 is a schematic diagram of a neural network according to various embodiments of the present application.

[0027] FIG. 3 is a configuration diagram of a terminal device of an airport bus according to various embodiments of the present application.

[0028] FIG. 4 is a flowchart of a method for connecting checked baggage of an airport bus according to various embodiments of the present application.

[0029] FIG. 5 is a detailed flowchart of a baggage image analysis process according to various embodiments of the present application.

[0030] FIG. 6 illustrates an image of a piece of luggage taken from inside a luggage bag according to various embodiments of the present application.

[0031] FIG. 7 illustrates the results of segmenting clothing items included in a luggage bag according to various embodiments of the present application.

[0032] FIG. 8 is a model network structure diagram of a second deep learning-based recognition model according to various embodiments of the present application.

[0033] FIG. 9 is a flowchart of the learning process of a second deep learning-based recognition model including a first encoder and a decoder according to specific embodiments of the present application.

[0034] FIG. 10 is a drawing illustrating checked baggage restrictions according to various embodiments of the present application.

[0035] FIG. 11 is a schematic diagram of a baggage linkage processing message according to various embodiments of the present application.

[0036] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0037] However, this disclosure is not intended to limit the present disclosure to a specific embodiment, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0038] In this specification, expressions such as “has,” “may have,” “includes,” or “may include” indicate the presence of a feature (e.g., a component such as a number, function, operation, step, part, element, and / or component), and do not exclude the presence or addition of additional features.

[0039] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0040] The terms "first," "second," "first," or "second" used in various embodiments may describe various components, regardless of order and / or importance, and do not limit the components. These terms may be used to distinguish one component from another. For example, "first component" and "second component" may represent different components, regardless of order or importance.

[0041] The embodiments of the singular expression used in this specification also include embodiments of the plural expressions, unless the phrases related to the singular expression clearly indicate a contrary meaning.

[0042] The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a device configured to" can mean that the device, together with other devices or components, is "capable of." For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0043] Terms, including technical or scientific terms, used in the present invention may have the same meaning as commonly understood by those of ordinary skill in the technical fields described herein. Terms defined in general dictionaries used in the present invention may be interpreted as having the same or similar meaning in the context of the relevant technology. Unless explicitly defined herein, they shall not be interpreted in an idealized or overly formal sense. In some cases, even if a term is defined herein, it cannot be interpreted to exclude embodiments of the present invention.

[0044]

[0045] The system (1, hereinafter referred to as the checked baggage linking system) for linking checked baggage on an airport bus using AI-based recognition and GPS coordinates is configured to link the airport bus and the airport security screening system by delivering the user's baggage carried by the airport bus from the user's boarding point to the airport stop to the airport security screening system installed at the airport.

[0046] Figure 1 illustrates a network environment of a system for linking checked baggage of an airport bus using AI-based recognition and GPS coordinates according to one aspect of the present application.

[0047] Referring to FIG. 1, the checked baggage linking system (1) may include a terminal device (100) of a user who must arrive at an airport to board an aircraft, a terminal device (200) of an airport bus on which the user will board to reach the airport, a server (300) that provides a checked baggage linking service, and an airport security inspection system (400) installed at the airport and performing the task of loading checked baggage onto an aircraft.

[0048] The checked baggage linking system (1) according to the embodiments may be entirely hardware, or may be partially hardware and partially software. For example, a system may collectively refer to hardware equipped with data processing capabilities and operating software for driving the hardware. In this specification, terms such as "unit," "system," and "device" are intended to refer to a combination of hardware and software driven by the hardware. For example, hardware may be a data processing device including a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or another processor. Furthermore, software may refer to a running process, an emoticon object, an executable, a thread of execution, a program, etc.

[0049] The user's terminal device (100), the airport bus terminal device (200), the server (300), and the airport security inspection system (400) can be connected to each other through a telecommunication network.

[0050] The telecommunication network provides a wired / wireless telecommunication path through which components (100, 200, 300, 400) of the checked baggage linkage system (1) can transmit and receive data to each other.

[0051] The telecommunications network is not limited to a specific communication protocol and any appropriate communication method may be used depending on the implementation. For example, if the system is configured based on the Internet Protocol (IP), the telecommunications network may be implemented as a wired and / or wireless Internet network. Alternatively, if the different devices (100, 200, or 300) are implemented as mobile communication terminals, the telecommunications network may be implemented as a wireless network, such as a cellular network or a wireless local area network (WLAN).

[0052]

[0053] The user's terminal device (100) and the airport bus terminal device (200) may be client terminal devices that communicate with the server (300).

[0054] The user's terminal device (100) and the airport bus terminal device (200) may be a computing system that includes hardware, software, or embedded logic components or a combination of two or more of these components and can perform appropriate functions implemented or supported by the device (100 or 200).

[0055] The terminal device (100) of the user and the terminal device (200) of the airport bus are devices including a processor, a memory, a communication unit, an input device, and an output device, and may be implemented in the form of a computer system, such as, for example, a desktop computer, a laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a portable electronic device, a cellular phone, a smart phone, other computing devices, other mobile devices, other wearable devices, other appropriate electronic devices, or any appropriate combination thereof.

[0056] The user's terminal device (100) may directly obtain mobile boarding pass data through the boarding pass purchase process, or indirectly obtain mobile boarding pass data by performing the boarding pass purchase process on another external device and transmitting the data. To this end, the user's terminal device (100) may access a website selling boarding passes (e.g., an airline, travel agency, or boarding pass sales platform) or install an application selling boarding passes.

[0057] The user's terminal device (100) may obtain mobile airport bus boarding pass data to be used by the user upon arrival at the airport to board an aircraft, or may indirectly obtain mobile airport bus boarding pass data by performing and transmitting the airport bus boarding pass purchase process from another external device. To this end, the user's terminal device (100) may access a website selling airport bus boarding passes (e.g., the airport bus homepage) or install an application selling airport bus boarding passes.

[0058] The user's terminal device (100) can pre-register an automatic payment method for purchasing airport bus tickets or airplane tickets for each user account on the system (1). The automatic payment method may be a credit card issued in the user's name or a simple payment service subscribed to in the user's name. The user's terminal device (100) can register the user's credit card as a payment method by entering information required for card payment, such as the card number, expiration date, and CVC, into the dedicated application of the system (1).

[0059] Or, when the user's terminal device (100) inputs the user's simple payment service account (e.g., user ID provided by the service) into the dedicated application of the system (1), the server (300) transmits a confirmation request to a third-party PG system providing the simple payment service, the third-party PG system issues a payment linkage token for the user and transmits it to the server (300), and the server (300) can register the payment linkage token for the user as the user's payment method. The payment linkage token is a token that has authentication information that grants payment authority. When the user attempts to pay for an airport bus ticket through the dedicated application, the server (300) notifies the third-party PG system that "a specific user is paying with this token" and requests payment, and the third-party PG system can approve the payment request based on the token.

[0060] Additionally, the user's terminal device (100) can store acquired airplane boarding pass data and airport bus boarding pass data.

[0061] Additionally, the user's terminal device (100) can capture a photo of the user's luggage to be loaded onto the aircraft to generate luggage bag image data. The luggage bag image represents the appearance of the luggage bag.

[0062] The user's terminal device (100) can capture the state of the luggage contained within the user's bag to generate luggage image data. The luggage depicted in the luggage image is expressed in a contained state. The luggage in the luggage image may include various pouches, shoes, and other travel items, including folded clothing, hats, underwear, accessories, socks, medicine, daily necessities, and / or small travel items. The luggage image may include luggage with an incomplete shape. For example, the luggage image may include luggage with an incomplete shape due to shape deformation, or luggage partially obscured by other luggage positioned above it. The luggage image is described in more detail with reference to FIG. 6 below.

[0063] Additionally, the user's terminal device (100) can analyze the user's baggage image and the user's baggage image to derive baggage data that the user wishes to check in. The baggage data includes item data obtained by analyzing the baggage image and the user's baggage data obtained by analyzing the user's baggage image.

[0064] The user's item data may include item information indicating the result of recognizing which product or product group the user's item belongs to, and weight information of the recognized item.

[0065] The above luggage bag data may include bag information (i.e., recognition information) of the luggage bag indicating the result of recognizing which bag product the user's luggage bag corresponds to, and weight information of the recognized luggage bag.

[0066]

[0067] The user's terminal device (100) and the airport bus terminal device (200) may include one or more deep learning models for analyzing various images. The deep learning models have a neural network structure.

[0068] FIG. 2 is a schematic diagram of a neural network according to various embodiments of the present application.

[0069] A neural network (or artificial neural network) can be composed of a set of interconnected computational units, generally referred to as nodes. These nodes can also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) that constitute a neural network can be interconnected by one or more links. Within a neural network, one or more nodes connected through links can form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, meaning that any node in an output node relationship to one node can also be in an input node relationship to another node, and vice versa. As described above, the input node-to-output node relationship can be created based on links. One input node can be connected to one or more output nodes through links, and vice versa.

[0070] In a relationship between input nodes and output nodes connected through a single link, the data of the output node can have its value determined based on the data input to the input node. Here, the link interconnecting the input nodes and output nodes can have a weight. The weight can be variable and can be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values ​​input to the input nodes connected to the output node and the weight set for the link corresponding to each input node.

[0071] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input and output node relationships within the network. The characteristics of a neural network can be determined based on the number of nodes and links, the relationships between nodes and links, and the weights assigned to each link. For example, if two neural networks have the same number of nodes and links but different weight values ​​for their links, the two neural networks can be perceived as different from each other.

[0072] A neural network can be composed of a set of one or more nodes. A subset of the nodes constituting the neural network can form a layer. Some of the nodes constituting the neural network can form a layer based on their distances from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links required to reach the node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the order of layers within a neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by its distance from the final output node.

[0073] An initial input node may refer to one or more nodes within a neural network to which data is directly input without going through links with other nodes. Alternatively, within a neural network, it may refer to nodes that do not have other input nodes connected by links in the relationship between nodes based on links. Similarly, a final output node may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, a hidden node may refer to nodes that constitute a neural network other than the initial input node or the final output node.

[0074] A neural network according to one embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be the same as the number of nodes in an output layer, and the number of nodes decreases and then increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be less than the number of nodes in an output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be greater than the number of nodes in an output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. A neural network according to another embodiment of the present disclosure may be a neural network in which the above-described neural networks are combined.

[0075] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a deep neural network, one can identify latent structures in data. For example, one can identify the latent structures of images, text, videos, audio, and music (e.g., what objects are in the image, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). Deep neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, generative adversarial networks (GANs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, and generative adversarial networks (GANs). The description of the deep neural network described above is only an example and the present disclosure is not limited thereto.

[0076] Neural networks can learn using at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Neural network learning can be the process of applying knowledge to the neural network to perform a specific action.

[0077] Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. In supervised learning, training data with the correct answer for each training data is used (i.e., labeled training data). In unsupervised learning, the correct answer may not be labeled for each training data. For example, in the case of supervised learning for data classification, the training data may be data in which each training data category is labeled. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the training data labels. Alternatively, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the neural network in the backward direction (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of neural network training to quickly achieve a certain level of performance, thereby increasing efficiency. A lower learning rate can be used in the later stages of training to increase accuracy.

[0078] In neural network training, training data can typically be a subset of real-world data (i.e., the data to be processed using the trained neural network). Therefore, there can be a learning cycle where errors on the training data decrease but errors on the real-world data increase. Overfitting is a phenomenon where excessive training on the training data leads to increased errors on the real-world data. For example, a neural network trained on yellow cats would fail to recognize cats when shown non-yellow colors, a phenomenon that could be a form of overfitting.

[0079] Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. These include increasing the training data, regularization, dropout (inactivating some network nodes during the learning process), and the use of batch normalization layers.

[0080]

[0081] To analyze an image, the user's terminal device (100) may include one or more deep learning models. The user's terminal device (100) may analyze the image using a specific deep learning model.

[0082] In various embodiments of the present application, the user's terminal device (100) may include a first deep learning-based recognition model that is machine-learned to recognize what kind of baggage product the baggage shown in the baggage bag image is.

[0083] A deep learning-based recognition model is a machine learning model that has a deep learning neural network structure and parameters learned to recognize objects.

[0084] The above first deep learning-based recognition model is configured to receive an image of the user's luggage as input data, and output bag information corresponding to the luggage bag shown in the input luggage bag image as output data.

[0085] In some embodiments, the first deep learning-based recognition model may have a similar configuration to a deep learning-based identification model. The first deep learning-based recognition model may have various neural network structures capable of recognizing objects in an input image. For example, the first deep learning-based recognition model may have a convolutional neural network (CNN) or other structure.

[0086] Additionally, in some embodiments, if the first deep learning-based recognition model fails to determine bag information corresponding to the luggage bag, i.e. fails to recognize what product the luggage bag is, the first deep learning-based recognition model may be further configured to determine the bag type to which the luggage bag belongs as bag information and output the bag type determination result as bag information. The first deep learning-based recognition model may be configured to determine the bag type and product value for the luggage bag. If the product value is not determined, the first deep learning-based recognition model may output a bag type recognition value and a value indicating a failure to recognize the product.

[0087]

[0088] In various embodiments of the present application, the user's terminal device (100) may include a machine-learned deep learning-based segmentation model to perform a segmentation operation that identifies and defines a sub-area (hereinafter, item area) in which baggage appears in a baggage image.

[0089] A deep learning-based segmentation model is a machine learning model that has a deep learning neural network structure and parameter values ​​learned to segment specific object regions in an input image.

[0090] The above deep learning-based segmentation model can be machine-learned to perform semantic segmentation, instant segmentation, or panoptic segmentation operations.

[0091] Semantic segmentation is the task of segmenting objects into distinct categories. In some embodiments, semantic segmentation is implemented by classifying each pixel in the target image into a class. In the semantic segmentation results, identical objects may be represented by the same color.

[0092] Instance segmentation is a task that segments objects individually. In some embodiments, instance segmentation can be implemented by labeling pixels within a region of interest (ROI) rather than labeling all pixels in the target image. The instance segmentation results are masked for each object.

[0093] Panoptic segmentation is a type of segmentation that combines semantic and instance segmentation. Panoptic segmentation segments an image using both instances (i.e., objects) and classes, labeling pixels based on instances of each class.

[0094] Deep learning-based segmentation models can have various neural network structures capable of segmenting object regions from input images. For example, the deep learning-based segmentation model may be R-CNN (Regions with Convolutional Neural Networks), Faster R-CNN, YOLO (You only look once), FCN, SSD (Single shot detector), or other segmentation models.

[0095] The above R-CNN is the first model to apply deep learning to the field of object detection, and is a two-stage model that separately deploys the region proposal and object detection stages. YOLO is a detection deep learning model that provides a new object detection methodology by defining object detection as a regression problem, going beyond the dimension of modifying a classification model, and obtaining bounding box coordinates and the probability of each class. Unlike the R-CNN model, which deploys the region proposal and object detection stages separately, SSD is a detection deep learning model that performs region proposal and object detection simultaneously without separating them.

[0096] FCN has a structure that is a modified CNN-based model, and unlike the basic VGG model that attaches a fully connected layer to the back of the network to extract image features, its structural characteristic is that CNN is placed at the back of the network instead of a fully connected layer for segmentation. ParseNet is a segmentation deep learning model that has received high evaluations in the PASCAL-Context competition and is a model that partially modifies the convolutional layers of FCN. DeconvNet (Convolutional and Deconvolutional Networks) is a segmentation deep learning model that combines the convolutional network and the inverse convolutional network of the VGG16 architecture. U-Net is a model that shows excellent performance even with a small amount of training data, and is a segmentation deep learning model with a U-shaped data processing path.

[0097]

[0098] In various embodiments of the present application, the user's terminal device (100) may include a second deep learning-based recognition model that is machine-learned to recognize what kind of product an item appears in an image of luggage is.

[0099] A deep learning-based recognition model is a machine learning model that has a deep learning neural network structure and parameters learned to recognize objects.

[0100] In some embodiments, the second deep learning-based recognition model may have a similar configuration to the first deep learning-based recognition model.

[0101] The second deep learning-based recognition model is configured to receive the user's baggage image as input data and output baggage information describing items appearing in the input baggage image as output data.

[0102] The above baggage information may be product information indicating items contained in a bag. Product information is information used to identify items for commercial purposes. In some embodiments, the product information may include one or more of the following: product group, product type, product name, product number, and manufacturer.

[0103] The second deep learning-based recognition model may have various neural network structures capable of recognizing objects in input images. For example, the second deep learning-based recognition model may have a convolutional neural network (CNN) or other structure.

[0104] In various embodiments of the present application, the second deep learning-based recognition model may be a ViT model.

[0105] Additionally, in some embodiments, if the second deep learning-based recognition model fails to determine product information corresponding to the baggage, i.e., fails to recognize what product the item is, the second deep learning-based recognition model may be further configured to determine the item type to which the baggage belongs as product information and output the result of determining the item type as product information. The second deep learning-based recognition model may be configured to determine the item type and product value for the baggage. If the product value is not determined, the second deep learning-based recognition model may output a recognition value of the item type and a value indicating a failure to recognize the product.

[0106] In various embodiments of the present application, the second deep learning-based recognition model may be further configured to recognize which product and / or which item type (e.g., product group) an imperfectly shaped item appears in the baggage image corresponds to. The second deep learning-based recognition model can recognize the baggage regardless of whether the shape of the baggage appears in the baggage image is complete or incomplete.

[0107] The above second deep learning-based recognition model can be machine-learned to recognize product information of baggage providing an incomplete shape by inferring a potential correlation between the features of baggage with an incomplete shape shown in the baggage image and product information of baggage providing the incomplete shape.

[0108] The second deep learning-based recognition model is described in more detail with reference to Figures 5 and 8 below.

[0109]

[0110] The user's terminal device (100) can calculate weight information of the recognized baggage based on the baggage bag recognition results. The user's terminal device (100) can calculate weight information of each recognized baggage item based on the baggage item recognition results having a complete or incomplete shape within the bag's interior image.

[0111] The user's terminal device (100) can generate user baggage data including bag information of the baggage, weight information of the baggage, baggage item information of each baggage item within the baggage, and weight information of each baggage item. The baggage data can describe a list of baggage items.

[0112] The user's terminal device (100) can transmit an image of the user's baggage to be loaded onto the aircraft, an image of the inside of the user's bag, and the user's baggage data to the server (300).

[0113] The operation of the user's terminal device (100) is described in more detail with reference to FIG. 4 below.

[0114]

[0115] FIG. 3 is a configuration diagram of a terminal device of an airport bus according to various embodiments of the present application.

[0116] Referring to FIG. 3, the terminal device (200) of the airport bus may include a communication unit (201), a processor (202), a memory (203), a GPS sensor (204), and a camera (205). In addition, in some embodiments, the terminal device (200) of the airport bus may further include a scanner (206). In addition, in some embodiments, the terminal device (200) of the airport bus may further include an input device (207) and an output device (208).

[0117] The communication unit (201) is configured to allow the terminal device (200) of the airport bus to transmit / receive data through wired / wireless electrical communication with another external device (e.g., the user's terminal device (100) or server (300)).

[0118] The communication unit (201) may include, for example, a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module. The wireless communication unit (160) may establish a communication connection with the user's terminal device (100) or server (300) via a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN).

[0119] The processor (202) may be configured to implement the procedures and / or methods proposed in the present invention.

[0120] The processor (202) may be implemented as at least one processor. The processor may include any combination of a CPU, a GPU (graphical processing unit), a single-core processor, a multi-core processor, an ASIC (application specific integrated circuit), and the like. In addition to a hardware implementation, the processor may be implemented as software and / or firmware. The software or firmware implementation of the processor may include computer- or machine-executable instructions written in any suitable programming language to perform the various functions described above. The software implementation of the processor may be stored in whole or in part within the memory (203).

[0121] The processor (202) may be configured to control other components to implement certain operations in the airport bus-linked checked baggage handling method, receive data from other components, perform computational processing, and transmit the results to other components. For example, the processor (202) transmits or receives information, etc., via the communication unit (201). Additionally, the processor (202) records and reads data in the memory (203).

[0122] The memory (203) can store programs of instructions that can be loaded and executed on the processor (202) and data generated during the execution of these programs. Examples of programs and data stored on the memory (203) may include an operating system that controls the operation of hardware and software resources available in the airport bus-linked checked baggage handling system (1), a driver for interacting with hardware devices such as a user's terminal device (100), an airport bus terminal device (200), and a server (300), a communication protocol for exchanging data with other hardware devices such as the user's terminal device (100) and the server (300), and additional software applications.

[0123] The memory (203) may be volatile (such as RAM) or non-volatile (such as ROM or flash memory). The memory (203) may provide storage of computer-readable instructions, data structures, program modules, and other data. The computer-readable medium may include at least two types of computer-readable media, namely, computer storage media and communication media.

[0124] The GPS sensor (204) is a component that detects the GPS (global positioning system) location of the airport bus. The GPS sensor (204) receives signals transmitted from three or more GPS satellites to determine the locations of the satellites and the GPS sensor (204). By measuring the time difference between the signal transmitted from the GPS satellite and the signal received by the GPS sensor (204), the distance between the GPS satellites and the GPS sensor (204) can be calculated. At this time, the signal transmitted from the GPS satellite contains information about the location of the GPS satellite. Once the distance to at least three GPS satellites and the location of each GPS satellite are known, the location of the GPS sensor can be calculated using a method such as trilateration. The terminal device (200) of the airport bus can determine the GPS location of the airport bus using the GPS sensor (204).

[0125] The GPS sensor (204) may be implemented in the form of a transmitter, receiver, or transceiver that communicates electrically using a protocol specialized for communication with GPS satellites.

[0126] The GPS sensor (204) can transmit GPS data having the GPS location information and detection time of the airport bus to the processor (202). The processor (202) can perform an aircraft boarding intent authentication operation based on GPS coordinates based on the GPS data having the GPS location information of the airport bus.

[0127] The camera (205) is a component that captures the face of a user boarding an airport bus. The camera (205) may be installed in a location within the airport bus (e.g., an entrance / exit door) where it is easy to capture the user's face.

[0128] The above camera (205) may be a variety of imaging devices that recognize light and generate images.

[0129] The camera (205) can capture the user's face and transmit the generated user's face image data to the processor (202). The processor (202) can perform an AI-based recognition operation based on the user's face image.

[0130] The scanner (206) is a component that scans a two-dimensional image corresponding to the mobile airport bus boarding pass displayed on the screen of the user's terminal device (100) to obtain information on the mobile airport bus boarding pass of a user boarding the airport bus.

[0131] The above two-dimensional image may be a QR code, barcode, or other readable image.

[0132] The terminal device (200) of the airport bus can determine whether the user has the right to board the bus based on the reading result of the scanner (206).

[0133] The input device (207) is a component configured to receive a command related to a user's input. The input device (207) may include a touch unit or other input unit. The touch unit is a component that inputs a user command by utilizing a part of the user's body or another object as a pointing object. The touch unit may include, but is not limited to, a pressure-sensitive or electrostatic sensor. The other input units include, for example, buttons, keyboards, dials, switches, sticks, etc.

[0134] An output device (208) is a component that visually or audibly outputs information stored and / or processed in the terminal device (200). The visually output device (208) may include, but is not limited to, an LCD, an OLED, a flexible screen, etc. The auditory output device (208) may be, for example, a speaker, etc.

[0135] Although the input device (207) and the output device (208) are separated in FIG. 3, in many embodiments, the input device (207) and the output device (208) may be implemented as a single component to perform input reception and information output. For example, the input device (207) and the output device (208) may be a touch panel implemented as a touch screen that forms a screen and layer structure. Touch input is input by a pointing object (e.g., a user's body, a tool, etc.).

[0136] Each component (201 to 208) of the terminal device (200) of the airport bus is connected to each other so as to be able to transmit / receive data, but does not necessarily constitute a single physically integrated device. In various embodiments of the present application, the camera (205) is installed at a location having a suitable field of view for capturing the face of a user boarding an airport bus, the scanner (206) is installed at a location suitable for a user boarding an airport bus to place a two-dimensional image displayed on his / her terminal device (100) in the reading area of ​​the scanner (206), and the remaining components (201 to 204, etc.) may be implemented in a form included within a single device.

[0137] The terminal device (200) of the above airport bus can produce GPS data of the airport bus through a GPS sensor (204).

[0138] The terminal device (200) of the above airport bus can capture the face of the user boarding the airport bus through a camera (205) and generate image data representing the face image of the user.

[0139] The terminal device (200) of the airport bus can analyze the user's facial image through the processor (202) to derive the user's identity data, and authenticate the user's intention to board the aircraft based on the user's GPS data received from the terminal device of the identified user and the GPS data of the airport bus.

[0140] The terminal device (200) of the above airport bus can request and obtain the mobile aircraft boarding pass data of the authenticated user from the terminal device of the user through the processor (202) and the communication unit (201).

[0141] The terminal device (200) of the above airport bus can generate a baggage linkage code corresponding to the user's mobile aircraft boarding pass based on the user's baggage data of the mobile aircraft boarding pass data through the processor (202).

[0142] The terminal device (200) of the above airport bus can transmit a linkage service reception message including the baggage linkage code to the user's terminal device through the server (300) via the communication unit (201).

[0143] Additionally, to analyze the image, the terminal device (200) of the airport bus may include one or more deep learning models.

[0144] In various embodiments of the present application, the terminal device (200) of the airport bus may include a deep learning-based identity verification model that extracts features from a facial image of a user boarding the airport bus and verifies the user's identity based on the extracted features.

[0145] The above deep learning-based identity verification model is a machine learning model configured to recognize the identity information of a user whose face appears in an input image by inferring a potential correlation between features extracted from the user's facial image and the user's identity information.

[0146] In one example, a deep learning-based identity verification model may include an input layer, a feature extraction layer, a fully connected layer, and an output layer.

[0147] The input layer is a layer into which a face image is input, and may be configured to input, for example, an image of size 224x224 with RGB channels, but is not limited thereto.

[0148] The feature extraction layer is configured to extract low-dimensional features from the input image. The feature extraction layer may include, for example, multiple convolutional layers and / or pooling layers. Convolutional layers consist of filters and activation functions for converting the input image into a feature vector. The ReLU activation function is typically used. Pooling layers are used to reduce the size of the feature map extracted from the convolutional layer and to emphasize important features. The max pooling method is typically used.

[0149] The fully connected layer performs classification based on the feature vectors from the pooling layer. Dropout techniques are typically applied to these fully connected layers to prevent overfitting.

[0150] The output layer is the layer that ultimately calculates class probabilities that identify a user based on the computational results of the fully connected layer. Typically, the softmax function is applied to the output layer to calculate class probabilities.

[0151] The output data of the deep learning-based identity verification model output from the above output layer has a vector value containing a probability value for the corresponding user.

[0152] The above deep learning-based identity verification model can be implemented in the form of, for example, VGGNet, ResNet, InceptionNet, or other CNN models.

[0153] In one embodiment, the deep learning-based identity verification model may be implemented in the form of an on-device deep learning model operated by a processor (202) and memory (203) in a terminal device (200).

[0154] In another embodiment, the terminal device (200) of the airport bus may be configured to perform an identity verification operation via an API accessible to a deep learning-based identity verification model. The deep learning-based identity verification model may operate on an external device (e.g., a third-party identity verification service provider).

[0155] The terminal device (200) of the above airport bus can verify a user's identity using a deep learning-based identity verification model. The user's identity verification information, obtained through the identity verification process, may include one or more of the user's name, gender, age, address, phone number, email address, profile image, and other profile information.

[0156] When an airport bus arrives at the airport, the user's baggage baggage, which has been assigned a baggage linkage code by the terminal device (200) of the airport bus, is delivered to the baggage handling area where the airport security inspection system (400) installed at the arriving airport is implemented.

[0157] The operation of the terminal device (200) of this airport bus is described in more detail with reference to Fig. 4 below.

[0158]

[0159] The employee terminal device (600) is a mobile terminal device that can communicate with the server (300) while being carried by the employee at the boarding airport.

[0160] The employee who provides user input on the employee terminal (600) is the employee who is in charge of checked baggage from the airport bus to the airport security inspection system inside the boarding airport after the airport bus arrives at the boarding airport during the airport bus-linked checked baggage handling process.

[0161] The employee who provides user input on the employee terminal (600) is the employee who is in charge of checked baggage from the airport bus to the airport security screening system (400) inside the boarding airport after the airport bus arrives at the boarding airport during the checked baggage handling process, and / or from the airport security screening system (400) to the aircraft.

[0162] The employee terminal (600) may be configured to receive the user's baggage tag information from the server (300) and print a label sticker representing the baggage tag. Furthermore, the employee terminal (600) may be configured to receive a baggage bag image representing the baggage to which the baggage tag is attached, along with the baggage tag information. The employee terminal (600) may be configured to display the baggage tag information and the baggage bag image.

[0163] The above employee terminal device (600) is implemented in a form that is connected to an external printer device (600-1) via wired / wireless connection, as illustrated in FIG. 1, or includes a print unit within the device. To this end, the employee terminal device (600) may include some or all of a processor, memory, communication unit, printer, I / O interface, input device, and output device. Since these components are similar to those of FIG. 3, a detailed description thereof will be omitted.

[0164]

[0165]

[0166] The server (300) is configured to perform a checked baggage linking operation to transfer the user's baggage loaded onto the airport bus upon boarding, to the airport security screening system (400) within the airport, after the user's intention to board the aircraft has been verified. To this end, the server (300) may be connected to the user's terminal device (100), the terminal device (200) of the airport bus, and the airport security screening system (400) within the airport for electrical communication.

[0167] In certain embodiments, the server (300) may be a unitary server or a distributed server spanning multiple computers or multiple data centers. In some embodiments, the server (300) refers to a computer system and computer software (network server program) that is connected to a sub-device capable of communicating with other network servers via a computer network, such as a private intranet or the Internet, and that receives a request to perform a task, performs the task, and provides the result of the task. However, the server (300) should be understood as a broad concept that includes, in addition to the network server program, a series of applications running on the server and, in some cases, various databases built within it. The server (300) may be of various types, such as, without limitation, a web server, a news server, a mail server, a message server, an advertisement server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In certain embodiments, each server (300) may include hardware, software, or embedded logic elements, or a combination of two or more such elements, to perform the appropriate functions implemented or supported by the server (300).

[0168] In various embodiments, a server (not shown) may include one or more data stores that store the server's data. The data stores may be used to store various types of information. In some embodiments, the information stored in the data stores may be structured according to a specific data structure. Furthermore, in some embodiments, each data store may be a relational, columnar, relational, or other suitable database. Although this disclosure describes or illustrates a particular type of database, this disclosure contemplates any suitable type of database. In some embodiments, the server may provide an interface that allows for managing, retrieving, modifying, adding, or deleting information stored in the data stores.

[0169] In various embodiments of the present application, the server (300) may include a bag DB that stores information about a plurality of pieces of luggage; and an item DB that stores information about items to be included in a piece of luggage.

[0170] The bag DB can store information about each bag, such as the bag type, product, manufacturer, weight, and other bag information.

[0171] The item DB can store information about each baggage item, such as baggage type, product, manufacturer, weight, and other item information.

[0172] The above server (300) can receive a baggage bag image taken of the user's baggage bag from the airport security inspection system (400) and / or reading result data for an X-RAY image of the baggage bag.

[0173] The reading result data may include an identification result for at least one item contained in the user's luggage. The identification result is expressed as a product group or item type of the luggage identified in the luggage through X-ray. The product group or item type may correspond to the classification result of the class in the aforementioned second deep learning-based recognition model.

[0174] In addition, the server (300) may receive the user's baggage data from the user's terminal device (100). In addition, in some embodiments, the server (300) may further receive the user's baggage image and baggage image. The baggage data, etc. may be received in advance before receiving the baggage image captured by the user's baggage from the airport security inspection system (400) and / or the reading result data for the X-RAY image of the baggage.

[0175] Then, the server (300) may associate the reading result data for the baggage with the baggage linkage code of the user based on the matching result of the baggage bag image of the airport security screening system and the baggage bag image of the user and the matching result of the reading result data of the airport security screening system and the item data in the baggage data of the user, and may transmit a notification message to the terminal device of the user based on the message regarding the baggage inspection result of the user received from the airport security screening system.

[0176] The operation of this server (300) is described in more detail with reference to FIG. 4 below.

[0177] The airport security inspection system (400) is a system that processes a series of baggage processes for loading baggage onto an aircraft at the airport.

[0178] The above airport security inspection system (400) can be implemented by at least one component among a plurality of computing devices, a conveyor belt system, and a moving means.

[0179] It may include a baggage screening device, a security screening device that controls the baggage screening device and sees through the inside of a baggage bag at the baggage screening device, an output device that outputs the inspection result of the security screening device, and / or a reading device that reads the inspection result of the security screening device.

[0180] Security screening devices may be a variety of devices designed to detect or discover restricted checked baggage items inside checked bags by screening personnel or computer algorithms.

[0181] The security screening device provides screening source data to screening personnel or readers, which is used to generate reading results that detect or discover restricted checked baggage items.

[0182] For example, the security screening device may be, but is not limited to, any one of an X-RAY scanner, a CT scanner, a 3D CT scanner, an EDS (Explosive Detection Systems) device, an ETD (Explosive Trace Detection) device, a neutron scanner, a gamma ray scanner, a large cargo scanner, and a combination of two or more thereof.

[0183] X-RAY scanners can pass X-rays through the interior of luggage to create X-RAY images. X-RAY images are used to visually identify foreign or prohibited items (such as explosives or weapons).

[0184] CT scanners are an advanced version of X-ray technology that can be used to analyze luggage into 3D images.

[0185] EDS and EDT devices are devices that detect small particles and chemical components of explosives contained in baggage. EDS and EDT devices can be used as an aid to other security screening devices.

[0186] Large cargo scanners can be used to inspect not only checked baggage but also cargo on cargo aircraft.

[0187] The reading device may be configured to identify the item type (or product group) of the user's baggage contained within the baggage through various reading algorithms that analyze the inspection results. To this end, the reading device may be configured to detect checked baggage that falls within the user's baggage restrictions guidelines, using various image analysis software or deep learning-based restricted baggage detection programs known at the time of filing of this patent.

[0188] The above checked baggage restrictions guidelines may include information on baggage capacity restrictions and prohibited baggage. The guidelines have been previously described with reference to Figure 10, and a detailed description thereof will be omitted.

[0189] The airport security screening system (400) is configured to detect the user's checked baggage corresponding to restricted baggage from the user's baggage through various restricted baggage detection methods known at the time of filing of the present application. For example, the airport security screening system (400) may detect the user's checked baggage corresponding to restricted baggage from the user's baggage by utilizing the technology described in Prior Art 1 (Patent Registration Publication No. 10-2063859 (Airport Security Screening System and Method Based on AI and Deep Learning, published on February 11, 2020), but is not limited thereto. Alternatively, the baggage screening system (400) may generate reading result data according to a user input of an inspection staff member. The user input of the inspection staff member may indicate the location of the user's baggage item suspected of being restricted baggage on the security screening image.

[0190] The airport security inspection system (400) can transmit the results of the checked baggage inspection to the server (300) based on whether the user's checked baggage corresponding to the checked baggage restriction is detected in the user's baggage.

[0191] In various embodiments of the present application, the airport security inspection system (400) may generate a message for baggage inspection and transmit it to the server (300) when checked baggage corresponding to restricted checked baggage is detected in the user's baggage.

[0192] The above-mentioned elimination message includes the baggage linkage code of the baggage bag detected as restricted checked baggage, location information for temporary storage of the baggage bag that failed baggage inspection or a location where the user can visit to check the baggage, and a message indicating the baggage inspection failure. In some embodiments, the elimination message may further include information on the item detected as restricted checked baggage.

[0193] In addition, the airport security inspection system (400) can issue a baggage management number for the user's baggage if the user's checked baggage corresponding to the checked baggage restriction is not detected in the user's baggage, generate a baggage inspection pass message including the baggage management number for the bag, and transmit the pass message to the server (300).

[0194] It includes the baggage management number for the above user's baggage and the message indicating successful passage of baggage inspection.

[0195] In various embodiments of the present application, the airport security screening system (400) may be further configured to print a baggage tag including a baggage management number issued for attachment to the user's baggage.

[0196] The above baggage tag includes the user's airline boarding pass information and baggage management number.

[0197] In the baggage handling space where the airport security inspection system (400) is implemented, a baggage tag can be attached to the baggage bag.

[0198] The operation of this airport security screening system (400) is described in more detail with reference to Fig. 4 below.

[0199]

[0200] It will be apparent to those skilled in the art that the baggage linking system (1) may include other components not described herein to implement the embodiments. For example, the baggage linking system (1) may include other hardware elements necessary for the operations described herein, including input devices for data entry, output devices for printing or other data display, and driving-related components such as wheels and links.

[0201]

[0202] Another aspect of the present application, a method for linking checked baggage of an airport bus using AI-based recognition and GPS coordinates (hereinafter, “a method for linking checked baggage of an airport bus”) can be performed by the baggage linking system (1) described above.

[0203] FIG. 4 is a flowchart of a method for connecting checked baggage of an airport bus according to various embodiments of the present application.

[0204] Referring to FIG. 4, the method for connecting checked baggage of an airport bus includes a step (S110) of obtaining mobile airport bus boarding pass data for the airport bus that the user will use to arrive at the airport to board the aircraft, through the terminal device of the user who will board the aircraft with baggage.

[0205] The above mobile airport bus boarding pass data is used to express a two-dimensional image such as a barcode on the user's terminal device (100).

[0206] In addition, the method for linking checked baggage of the airport bus includes a step (S120) of taking a picture of the user's baggage to be loaded onto the aircraft by the user's terminal device (100) to generate baggage bag image data representing the baggage bag image, and taking a picture of baggage contained inside the user's baggage bag to generate baggage image data representing the baggage image; and a step (S130) of analyzing the taken image of the user's baggage bag and the user's baggage image to derive the user's baggage data; and a step (S190) of transmitting the user's baggage data to a server (300).

[0207] In the above step (S130), the user's terminal device (100) can analyze a baggage image or a baggage bag image using one or more deep learning models to generate the user's baggage data.

[0208] In the above step (S130), the user's terminal device (100) can analyze the baggage image using one or more deep learning models to generate the user's item data.

[0209] The above user's item data has information describing the baggage contained in the user's luggage bag.

[0210] The user's item data may include item information for each item (i.e., recognition information) indicating the result of recognizing which product or product group the user's item belongs to, and weight information of the recognized item.

[0211] FIG. 5 is a detailed flowchart of a baggage image analysis process according to various embodiments of the present application.

[0212] Referring to FIG. 5, the step of calculating the user's item data (S130) includes the step of recognizing the user's luggage by applying the image of the user's luggage to a first deep learning-based recognition model (S131); and the step of searching for weight data of the user's luggage through the Internet or a server based on the result of the recognition of the luggage.

[0213] In the above step (S131), the luggage bag recognition result indicates the product or type of luggage bag shown in the user's luggage bag image. In some embodiments, the luggage bag recognition result may include one or more of the bag type, bag manufacturer, and bag product number of the corresponding bag product.

[0214] In the above step (S132), the user's terminal device (100) can transmit a search query to the Internet or server (300) based on the identification information of the luggage bag. The search query has the identification information of the luggage bag.

[0215] For example, if the result of recognizing a luggage bag from an image of a luggage bag is "American Tourister ROCKFORD69 Carrier BG941002", the user's terminal device (100) can search for information on the luggage bag through the bag DB of the Internet or server (300) and obtain the weight of the bag "3.96 kg" from the information on the luggage bag.

[0216] In addition, the step (S130) includes a step (S133) of applying the user's luggage image to a deep learning-based segmentation model to segment the item area where the luggage appears in the user's luggage image by each class of the luggage and extracting the segmented item area as a sub-image; a step (S135) of applying the sub-image of the extracted item area to a second deep learning-based recognition model to recognize the item appearing in the user's luggage image; a step (S136) of searching for weight data of the item in the user's luggage based on the recognition result of the item; and a step (S137) of calculating weight information of the user's luggage based on the weight search result of the recognized item in the recognized luggage. In some embodiments, the step (S130) may further include a step (S134) of calculating size information for each piece of luggage in the luggage based on the recognition result of the user's luggage and the segmentation result of the item area.

[0217] FIG. 6 illustrates an image of luggage captured from the inside of a luggage bag according to various embodiments of the present application. FIG. 7 illustrates the results of segmenting clothing items contained in a luggage bag according to various embodiments of the present application.

[0218] Referring to FIG. 6, the image of the user's luggage captured in step (S132) is in a state of being contained in a luggage bag. The items shown in the luggage image may be complete items without partial obscuration and / or incomplete items.

[0219] As illustrated in FIG. 6, the luggage image may include clothing (I1, I2, I4, I5) and a hat (I3) in a shape deformed to fit the interior space of the luggage bag.

[0220] Alternatively, the luggage image may show a fully formed item, such as a pair of sandals.

[0221] Incompletely shaped baggage may be partially obscured by other baggage, or may be a deformed version of a fully shaped baggage. As illustrated in Figure 6, a pouch may be partially obscured by sandals, resulting in an incomplete shape. Clothing such as skirts and hats may be folded or deformed, resulting in an incomplete shape.

[0222] In the above step (S133), the user's terminal device (100) can input a baggage image, such as that shown in FIG. 6, into a deep learning-based segmentation model. When the user's baggage image is input, the deep learning-based segmentation model can be machine-learned to identify the item type of the baggage shown in the baggage image by inferring a correlation between the features of the baggage and the item type of the baggage in the baggage image, and to identify the item area, which is a sub-area in which the baggage appears in the baggage image.

[0223] The above deep learning-based segmentation model can have various neural network structures capable of performing segmentation operations as described above.

[0224] The above deep learning-based segmentation model has a baggage image as an input image.

[0225] The above deep learning-based segmentation model outputs item regions from luggage images. The item regions are expressed as the result of labeling pixels in the luggage image where the corresponding item appears with the corresponding item's class.

[0226] In the above deep learning-based segmentation model, classes are determined by the segmentation method. For example, in a deep learning-based segmentation model that applies semantic segmentation as the segmentation method, the class may be an item type.

[0227] In various embodiments of the present application, the deep learning-based segmentation model may be trained using a first training data set. The first training data set comprises a plurality of training samples, each of which comprises training data and label data.

[0228] The above training data may be at least one training image.

[0229] At least one training image may be a sample image of an imperfect shape, representing a learning sample item (e.g., clothing) having an imperfect shape. In some embodiments, the imperfect shape represented in the at least one training image may be a shape according to a state in which the item is received in a luggage bag.

[0230] In some embodiments, the training data may further include a sample image of a complete shape, representing an article having a complete shape as the article shown in the sample image of the incomplete shape.

[0231] The label data represents the actual values ​​of the item types of luggage that appear in the training images.

[0232] When a segmentation model is trained using the first training data set, the segmentation model can learn to recognize an item with an incomplete shape and segment a sub-region in which an item with an incomplete shape appears in an input image based on features of a boundary portion of a complete item appearing in a sample image of a complete shape, features of a boundary portion of an item with an incomplete shape appearing in a sample image of an incomplete shape, and features of a difference in appearance between an incomplete shape and a complete shape of the same item.

[0233] In some embodiments, the deep learning-based segmentation model can learn to recognize items with imperfect shapes by giving more weight to features on the boundary portions of the items with imperfect shapes.

[0234] The deep learning-based segmentation model trained in this way is fed with images of luggage.

[0235] As a result, the deep learning-based segmentation model can accurately detect item areas where items with incomplete or complete shapes appear, as illustrated in Fig. 7.

[0236]

[0237] Additionally, in various embodiments of the present application, the server (300) may extract one or more sub-regions of an item region in which a complete-shaped item appears from an original image for a sample in which a complete-shaped item appears, and generate a first training data set having images of the extracted sub-regions as training images. The sub-regions represent a portion of the complete shape. Some of the one or more sub-regions extracted from the original image of the same item may represent different portions of the item from other sub-regions.

[0238] Using the deep learning-based segmentation model learned using this first training data set, the user's terminal device (100) can obtain the item area as a segmentation result and extract the item area from the baggage image.

[0239] As illustrated in Fig. 7, a deep learning-based segmentation model can be used to segment multiple item areas representing clothing luggage from a luggage image (S133).

[0240] In the above step (S133), the user's terminal device (100) can extract a sub-image including one or more item areas from a single baggage image based on the segmentation result.

[0241] The sub-image of the above item area is a sub-image including the segmented item area from the above luggage image, and is used as an input image to the second deep learning-based recognition model.

[0242] In various embodiments of the present application, the step (s133) of extracting the sub-image may include, before applying the second deep learning-based recognition model, a step of calculating a hierarchical relationship between items appearing in the segmented item areas; a step of selecting item areas in which items having a hierarchical relationship with respect to a specific item simultaneously appear and are separated from each other; a step of detecting attribute information of the selected item areas; a step of determining item areas representing the same item among the separated item areas based on the attribute information of the selected item areas as item areas representing the same item; and a step of extracting a single sub-image including the separated item areas.

[0243] The user's terminal device (100) can select adjacent item areas from among the divided item areas and determine the upper-lower relationship between adjacent items appearing in each adjacent item area based on one or more of the brightness distribution, the area of ​​the shadow portion, the shape of the shadow portion, and the shape of the item in the adjacent item area.

[0244] In the above step (s133), the user's terminal device (100) can analyze the superior-subordinate relationship between the floral skirt (I1) and the white top (I4) and determine that the floral skirt (I1) is positioned higher than the white top (I4).

[0245] In the above step (S133), the user's terminal device (100) selects different item areas (A4) representing white tops (I4), and analyzes and calculates attribute information of the two selected item areas (A4). The attribute information of the areas may include one or more of brightness, texture, color, and pattern.

[0246] The user's terminal device (100) can determine the item areas (A4) representing the same item, a white top (I4), among the item areas separated from each other based on the attribute information of the item as the same item area.

[0247] Then, the user's terminal device (100) can extract a single sub-image including the separated item areas (A4) as a sub-image for the white top (I4) (S133).

[0248] The above step (S134) is a step in which the user's terminal device (100) searches for size information about the user's luggage bag recognized in step (S131) ​​from the Internet or a server (300); and based on the size information about the user's luggage bag, the size information of the luggage bag area in the luggage image, and the size information of the segmented item area, the size information of the items contained in the luggage bag can be calculated. The size of the luggage bag area is the size of the sub-area segmented into the luggage bag area in the luggage image.

[0249] The size information of the item has width and height obtained through the bounding box of the baggage.

[0250] Referring again to FIG. 5, in step (S135), the user's terminal device (100) can input the sub-image extracted in step (S133) as an input image to the second deep learning-based recognition model.

[0251] Additionally, in some embodiments, in step (S135), the user's terminal device (100) may additionally input size information of the item shown in the sub-image produced in step (S134) into the second deep learning-based recognition model.

[0252] FIG. 8 is a model network structure diagram of a second deep learning-based recognition model according to some embodiments.

[0253] Referring to FIG. 8, the second deep learning-based recognition model may include an input layer (810), a preprocessing layer (820), a feature extraction layer (830), a fully connected layer (850), and an output layer (870). The input layer (810), the feature extraction layer (830), the fully connected layer (850), and the output layer (870) have been described above, and differences therebetween will be mainly described.

[0254] The input layer (810) is a first input layer, into which a sub-image of the item area extracted in step (S133) is input. The first input layer (810) may be configured to input, for example, an image of size 224x224 with RGB channels, but is not limited thereto.

[0255] Sub-image data input through the first input layer (810) is input to the feature extraction layer (830).

[0256] The preprocessing layer (820) is a second input layer and is configured to receive the size information of the product produced in step (S134) and normalize it based on the size of the sub-image.

[0257] In some embodiments, the second deep learning-based recognition model can produce normalized size information (w', h') of the corresponding baggage appearing in the item area through the following mathematical formula.

[0258] [Mathematical Formula 1]

[0259] w' = w / W

[0260] h' = h / H

[0261] Here, w and h represent the width and height of the baggage produced in step (S134), respectively, and W and H represent the height and size of the image.

[0262] The normalized size information of the baggage processed in the second input layer (820) is combined with the feature extraction result of the feature extraction layer (830).

[0263] The feature extraction layer (830) is configured to extract features from sub-images of the item area. The extracted features may be implemented in the form of a feature map. Information contained in the feature map represents features useful for recognizing items in a luggage bag with a condition (e.g., an incomplete shape).

[0264] In various embodiments of the present application, the feature map may include features extracted from the boundary portion of the baggage in the sub-image.

[0265] The above second deep learning-based recognition model combines the values ​​of the size information of the product normalized in the preprocessing layer (820) with the features (i.e., feature map) extracted in the feature extraction unit 830, and inputs the combined result into the fully connected layer (850).

[0266] In some embodiments, the second deep learning-based recognition model may be configured to convert the normalized size information of the item into a size information tensor, generate tensor data of the baggage based on the feature map and size information tensor extracted from the sub-image, and input the generated tensor data of the baggage into a fully connected layer (850) as an intermediate operation result.

[0267] If the size of the feature map is (C, Hf, Wf), the size information tensor can be expressed in the form of (2, w', h').

[0268] The second deep learning-based recognition model combines the size information tensor with the feature map of the sub-image at the channel level. The combined baggage tensor data then represents a final tensor with two additional channels (size information) added to the original number of channels in the feature map of the original sub-image.

[0269] The fully connected layer (850) is trained to receive and process tensor data of the luggage, thereby inferring potential correlations between the size information and feature information of the item and the condition (e.g., incomplete shape) of the luggage contained in the bag, thereby recognizing the product or product group of the item shown in the sub-image. The feature information may represent the characteristics of the luggage in an incomplete shape, such as partial occlusion or deformation.

[0270] Through the above fully connected layer (850), the second deep learning-based recognition model can recognize what kind of product or what kind of product group the item shown in the sub-image having the incomplete shape is by considering the characteristics of the baggage with the incomplete shape.

[0271] The above fully connected layer (850) can calculate the probability that the item in the sub-image corresponds to a certain product.

[0272] The above output layer (870) can output product recognition results based on the computational results of the fully connected layer (850), i.e., the inference results. The recognition results are output as product information or product group information. The product group may be a product type.

[0273] The recognition result may be product information of the actual baggage or product information of similar baggage recognized as most similar to the actual baggage.

[0274] In some other embodiments, the second deep learning-based recognition model may be a ViT model. The ViT (Vision Transformer) model is an application of the Transformer model, originally used in natural language processing, to image recognition. It transforms image patches into a series of tokens, which are then input into a Transformer encoder to learn relationships between patches.

[0275] The second deep learning-based recognition model includes an input layer that inputs the sub-image as an input image, an encoder including one or more encoding blocks, a decoder including one or more decoding blocks, and a classification layer that recognizes and outputs an item appearing in the sub-image based on an encoding vector output from the decoder.

[0276] The second deep learning-based recognition model may be configured to divide an input sub-image into a plurality of patches, convert each patch into a patch embedding vector, combine a position embedding value indicating the position of each patch with the patch embedding vector of the corresponding patch, input the combined vector into the encoder for processing to produce an encoding vector, recognize an item appearing in the sub-image based on the encoding vector, and output a recognition result.

[0277] In the ViT model, the input sub-images can be multiple smaller-sized patch images (e.g., 16x16 pixels). Each patch is converted into a fixed-length vector, which has a form similar to tokenized text.

[0278] Through patch embedding, each patch vector is mapped to a high-dimensional space, and location information of the patch is reflected in the patch vector by adding a location encoding that indicates the location of the patch in the sub-image.

[0279] In the ViT model, the transformer encoder receives an input sequence combining patch vectors and position encodings. The encoder learns relationships between patches through a multi-head self-attention mechanism. This process assigns greater weight to patches that are relatively important for recognizing objects in sub-images, thereby learning to overcome imperfect geometry, such as partial occlusion, and recognize objects.

[0280] Specifically, a transformer encoder can be implemented by stacking multiple encoder layers. The encoder layers can be composed of self-attention and feedforward layers. Self-attention is used to determine the relationship between each word, and each word independently calculates weights to determine how they influence each other. The embedding vector input to the prompt transformation model is input to the first encoding layer of the encoder.

[0281] In some embodiments, the encoder layer may consist of a multi-head attention layer and a feed forward layer.

[0282] Multi-head attention is divided into multiple attention heads, each capable of extracting specific information from a given sequence. If there are h attention heads, each token in the token embedding is analyzed into h different pieces of information. This analysis process is similar to the role of filters in a convolutional neural network in image processing. Analyzing h different pieces of information is similar to how one filter in a convolutional neural network processes circles and another processes squares. One attention head calculates a weight matrix and then calculates a weighted average for each token in the token embedding. This weight matrix can be referred to as an attention score. Through this process, a similarity score is calculated between each token in the token embedding. Tokens with high correlation receive high scores, while tokens with low correlation receive low scores. By increasing the scores of the parts to be emphasized, the attention mechanism is applied, and a self-attention operation is performed to calculate scores for the same token embeddings.

[0283] The attention head can be implemented by creating and calculating the query vector (Q), key vector (K), and value vector (V) through a fully connected layer with token embeddings. The weight matrix is ​​created by taking the dot product (or MatMul) of the query vector and the key vector, normalizing this matrix and applying the softmax function so that the column sum becomes 1, and then multiplying the weight matrix and the value vector to produce the final result. Meanwhile, when learning to predict recommended prompts, an additional operation is performed before the softmax function of the weight matrix to prevent the current token from referencing the following tokens (i.e., tokens representing the recommended prompt) by processing the positions of the following tokens as negative infinity. This additional operation can be referred to as masking. By stacking multiple of these attention heads, a multi-head attention can be created.

[0284] The feed-forward layer consists of multiple fully connected layers. For example, the feed-forward layer may consist of two fully connected layers.

[0285] In the above ViT, the encoder transformer outputs class tokens. The class tokens represent summary information about the image. The class tokens of the ViT are input to the classification layer.

[0286] The above classification layer recognizes what kind of product an item is or what kind of product group it is based on the class token.

[0287]

[0288] In various embodiments of the present application, the second deep learning-based recognition model may be trained using a second training data set. The second training data set comprises a plurality of training samples, each of which comprises training data and label data.

[0289] The above training data may be at least one training image.

[0290] At least one training image may be a sample image of an imperfect shape, representing a learning sample item (e.g., clothing) having an imperfect shape. In some embodiments, the imperfect shape represented in the at least one training image may be a shape according to a state in which the item is received in a luggage bag.

[0291] In some embodiments, the training data may further include a sample image of a complete shape, representing an article having a complete shape as the article shown in the sample image of the incomplete shape.

[0292] The label data represents the actual values ​​of the item types of luggage that appear in the training images.

[0293] When training a recognition model using the second training data set, the recognition model can learn to recognize an item with an incomplete shape by using features of the boundary portion of a complete item appearing in a sample image of a complete shape and features of the boundary portion of an item with an incomplete shape appearing in a sample image of an incomplete shape.

[0294] In some embodiments, the second deep learning-based recognition model can learn to recognize objects with imperfect shapes by giving more weight to features of the boundary portions of objects with imperfect shapes.

[0295] The sub-images of the item area are input to the second deep learning-based recognition model trained in this way.

[0296] As a result, the second deep learning-based recognition model can accurately detect the item area in which an item with an incomplete shape or a complete shape appears in Fig. 6.

[0297]

[0298] In various embodiments of the present application, the feature extraction layer (830) in the second deep learning-based recognition model of FIG. 8 may include a first encoder (831) and a decoder (835). The first encoder (831) and the decoder (835) may be trained by interacting with the second encoder (832). The first encoder (831) corresponds to an encoder in ViT, and the decoder (835) corresponds to a decoder in ViT.

[0299] FIG. 9 is a flowchart of the learning process of a second deep learning-based recognition model including a first encoder and a decoder according to specific embodiments of the present application.

[0300] Referring to Figure 9, the second training data set consists of multiple training samples. Each training sample includes a pair of sample images.

[0301] The above pair of sample images is a sample image of an incomplete shape, representing a learning sample item (e.g., clothing) with an incomplete shape, and a sample image of a complete shape, representing a complete shape of the same item.

[0302] The above-mentioned sample image of a complete shape may be an image (e.g., a frontal or side image of a product) provided by a seller for sale in an online shopping mall or smart store. For example, it may be a product image of a non-foldable garment (e.g., I1, I2). Alternatively, it may be a product image of a non-wrinkle garment accessory (e.g., I3).

[0303] The sample image of the above imperfect shape shows a shape identical or similar to the shape according to the state in which it is received in the luggage bag.

[0304] The sample image of the above incomplete shape may be a sample image actually captured in a state of being contained in a luggage bag, or may be an image determined by dividing the sample image of the above complete shape into a plurality of sub-regions and selecting some of the sub-regions. The plurality of sub-regions may be, for example, rectangular regions defined by dividing the image at regular intervals, but are not limited thereto.

[0305] Partitioning a sample image into multiple sub-regions may be referred to as Patchify.

[0306] Some areas selected as sample images correspond to areas in which the incompletely shaped sample images are actually contained in the luggage bag. In some embodiments, some areas selected as sample images may be two or more adjacent areas. This is because even if the items contained in the luggage bag are partially obscured or wrinkled, some parts of the body appear as continuous, complete shapes.

[0307] The second encoder (832) is configured to process a sample image of a complete shape among a pair of sample images to extract features of an article of a complete shape.

[0308] The first encoder (831) is configured to process a sample image of an incomplete shape among a pair of sample images to extract features of an item of an incomplete shape.

[0309] The output of the first encoder (831) may be a feature set masked from the feature set (e.g., a feature map or vector) of the second encoder (832). Features corresponding to sub-regions that are incompletely represented in the feature set of the second encoder (832) may be masked in a form in which they are removed. The output of the first encoder (831) includes position embeddings of sub-regions corresponding to sample images of incomplete shapes input to the first encoder (831) among the segmented sub-regions.

[0310] The decoder (835) may include multiple decoder blocks. The decoder (835) may be configured to implement an attention mechanism. The output of the second encoder (832) may be used as a key and a value in the attention mechanism, and the output of the first encoder (831) may be used as a query (q) in the attention mechanism.

[0311] Specifically, each decoder block may include a cross-attention layer and a self-attention layer. The output of the first encoder (831) is attended to the output of the second encoder (832) through the cross-attention layer. Then, through the self-attention layer, the output of the first encoder (831) and the output of the second encoder (832) are attended to each other.

[0312] The output sequence of the above decoder (835) is a feature sequence that can be used to predict pixel values ​​for a sample image of a complete shape from a sample image of an incomplete shape. The decoder (835) is trained to restore features that can be used to predict a complete shape from a sample image of an incomplete shape and to implement an image of an item of the complete shape. The decoder (835) that has completed training can restore and output features that are identical or very similar to the features extracted from the sample image of the complete shape.

[0313] The features restored by the output sequence of the decoder (835) are features that can be extracted from the predicted complete shape of the product image. By inputting the output sequence of the decoder (835) into an image generation AI model, an image of the predicted complete shape of the product can be generated.

[0314] The output sequence of the decoder (835) is input to the fully connected layer (850).

[0315] The above fully connected layer (850) can classify which of the preset products corresponds to an item appearing in an incomplete sample image based on the features restored by the decoder (835). The item appearing in the incomplete sample image is recognized as a classified product.

[0316] The model components (810, 832, 831, 835, 850) of FIG. 9 can be learned through various learning methods. For example, the model components (810, 832, 831, 835, 850) can be learned through self-supervised learning.

[0317]

[0318] In various embodiments of the present application, the second deep learning-based recognition model may be a foundation model pre-trained on a specific data set, fine-tuned to recognize items of imperfect or complete shapes contained within a luggage bag. In some embodiments, after fine-tuning, the second deep learning-based recognition model may be further trained to recognize items from images of items of imperfect shapes.

[0319] Fine-tuning can be performed by adjusting the weights of the entire second deep learning-based recognition model, or by adjusting a partial network. The partial network adjustment can be performed by adjusting the conversion performance of the encoder's sublayers (e.g., the initial layer, the final layer) to a low-dimensional feature space in the feature extraction layer (830) and / or at least some of the weights of the fully connected layer (850).

[0320] The above foundation model can have various ViT-based neural network architectures. For example, the above foundation model can be implemented using the Segment Anything Model (SAM), Segment SAM, or a neural network structure utilizing the same.

[0321] In various embodiments of the present application, the foundation model may be pre-trained through the learning process of FIG. 9. In this case, the foundation model fine-tuned to be used as the second deep learning-based recognition model may include some or all of the components (810, 820, 832, 831, 835, 850) of FIG. 9.

[0322] For example, the foundation model may include an input layer (810), a second encoder (832), a first encoder (831), a decoder (835), a fully connected layer (850), and an output layer (870). The second deep learning-based recognition model fine-tuned from the foundation model may include an input layer (810), a first encoder (831), a decoder (835), a fully connected layer (850), and an output layer (870). The first encoder (831), the decoder (835), and the fully connected layer (850) of the foundation model and the first encoder (831), the decoder (835), and the fully connected layer (850) of the fine-tuned second deep learning-based recognition model may have different model parameters (e.g., weights) or settings.

[0323]

[0324] In alternative embodiments, the second deep learning-based recognition model can be configured and trained to recognize an item appearing in a sub-image without using size information of the item.

[0325] Referring again to FIG. 5, in the step (S136), the user's terminal device (100) can retrieve weight data of each piece of luggage recognized in the luggage image through the Internet or a server (300) based on the recognition results of individual items (e.g., product information).

[0326] The process of searching the weight of the baggage in the above step (S136) is similar to the process of searching the weight of the baggage in step (S132), so a detailed description is omitted.

[0327] In the above step (S137), the user's terminal device (100) can calculate weight information of the user's baggage based on the weight search result of the recognized baggage searched in step (S132) and the weight search result of the recognized item searched in step (S136).

[0328] The weight information of the above baggage is calculated as the total weight of the baggage by adding the weight information of the baggage bag and each recognized item.

[0329] In some embodiments, the step (S137) of calculating the user's item data may, if the recognized item is a pouch, predict the weight of the entire pouch by considering the weight of the items contained in the inner space of the pouch, and calculate weight data of the entire baggage including the pouch based on the predicted result.

[0330] Specifically, the step (S137) may further include a step of requesting input of information on items inside the pouch included in the pouch when the class of the recognized item corresponds to a pouch; and a step of retrieving weight data on the pouch based on the information on the items inside the pouch input and the size data of the pouch calculated from the segmentation result of the item area in which the pouch appears.

[0331] In the above step (S137), the user's terminal device (100) may display a GUI screen that prompts the user to input product information about the product inside the pouch. The user input may be text input, click, touch, or other selection input.

[0332] The user's terminal device (100) determines, based on product information of the product inside the pouch and size data of the pouch, baggage that has product information and is smaller than the size of the pouch and can be accommodated in the internal space as the product inside the pouch, and can search for weight information of each of the determined product inside the pouch.

[0333] The user's terminal device (100) can calculate the weight of the entire pouch by adding up the weight of each item inside the pouch, and ultimately generate more accurate item data.

[0334] Additionally, in some embodiments, the baggage data may further include a baggage list. The baggage list includes product information and corresponding weight information for each piece of baggage, as well as product information and corresponding weight information for each item.

[0335] The weight information of the above-mentioned baggage can be stored in the user's terminal device (100). In addition, the user's terminal device (100) can transmit the weight information of the user's baggage to the server (300) and store it (S137).

[0336] Additionally, in various embodiments of the present application, the step (S130) may further include a step (S138) of detecting restricted checked baggage among the recognized items; and a step (S139) of providing a warning message including the detection result of restricted checked baggage to the user.

[0337] FIG. 10 is a schematic diagram of checked baggage restriction information according to various embodiments of the present application.

[0338] Referring to FIG. 10, the user's terminal device (100) can store restricted baggage information. The restricted baggage information is a guideline that restricts the consignment of baggage. The restricted baggage information may include prohibited baggage information and capacity-limited baggage information, as illustrated in FIG. 10.

[0339] In the above step (S138), the user's terminal device (100) can search for baggage corresponding to restricted checked baggage among the recognized items and create a list of restricted checked baggage.

[0340] In the above step (S139), the user's terminal device (100) can output a warning message audibly or visually notifying that checked baggage is included in the baggage bag.

[0341] In various embodiments of the present application, the step (S190) receives a user input indicating the fact of removal of the user's checked baggage, and then calculates the weight of the user's baggage from the weight information of the user's baggage calculated in the step (S137).

[0342] Referring again to FIG. 4, in step (S190), the server (300) can store the received user's baggage data.

[0343] In some embodiments, the server (300) may store the user's luggage bag image and luggage image in association with the luggage data (S190).

[0344] In addition, the method for connecting checked baggage of the airport bus includes some or all of the processes (S210 to S290) for authenticating the intention of a user boarding the airport bus to board an aircraft based on GPS.

[0345] Specifically, the method for linking checked baggage of the airport bus may include a step (S220) of capturing a face image of a user boarding the airport bus by a terminal device (200) of the airport bus; a step (S230) of analyzing a face image of the user boarding the airport bus by capturing the face image of the user by the terminal device (200) of the airport bus to verify the identity of the user; a step (S250) of authenticating a user's intention to board an aircraft based on GPS data of the user and GPS data of the airport bus received from a terminal device of the user whose identity has been verified by the terminal device of the airport bus; a step (S270) of requesting and obtaining mobile boarding pass data for an aircraft of the user whose intention to board the aircraft has been verified by the terminal device of the user, generating a baggage linkage code corresponding to the mobile boarding pass of the user based on item data of the user boarding the airport bus, and generating the baggage linkage code; and a step (S290) of transmitting a result of the user's identity verification and a baggage linkage code to a server by the terminal device of the airport bus. In some embodiments, the checked baggage linking method may further include a step (S210) of reading the user's mobile airport bus boarding pass to confirm whether the user has the mobile airport bus boarding pass.

[0346] In the above step (S210), the scanner (206) scans a two-dimensional image corresponding to the mobile airport bus boarding pass and reads the mobile airport bus boarding pass data included in the two-dimensional image, thereby confirming whether or not the user boarding the airport bus is in possession of the mobile airport bus boarding pass.

[0347] In the above step (S230), the terminal device (200) of the airport bus can acquire the user's identity information by inputting the user's facial image captured in step (S220) into a pre-trained identity verification model. The user's identity information may include name, date of birth, phone number, address, profile image, and / or other profile information.

[0348] In various embodiments of the present application, the step (S230) of obtaining the user's identity information may include: a step of obtaining the user's identity information by analyzing the user's facial image; a step of confirming whether a user with the obtained identity information is included in a bus passenger list obtained in advance from a server (300); and a step of finally obtaining the identity information only for users included in the bus passenger list.

[0349] The user can input the location information of the stop at which he or she will board the airport bus and the boarding time information into the system in advance through his or her terminal device (100).

[0350] The server (300) can create a bus passenger list consisting of each user's stop and boarding time information based on the user's stop and boarding time information, and provide the list to the terminal device (200) of the airport bus.

[0351] The terminal device (200) of the airport bus captures a user's facial image at a specific stop and time, and primarily analyzes the user's identity information from the captured facial image. The system (1) may be configured to use the analyzed identity information only if the user whose identity information has been analyzed is included in the list of users scheduled to board the bus at the specific stop and time.

[0352] On the other hand, the system (1) may be configured not to use the analyzed identity information if the user whose identity information has been analyzed does not belong to the list of users scheduled to board the bus at the specific stop and at the specific time in the list of bus passengers.

[0353] A user's facial information constitutes personal information. The system (1) above is configured to analyze facial images and use them for identification purposes only for users who have consented to the use of their personal information by entering information on the location of the airport bus stop and the boarding time into the system.

[0354] The user's terminal device (100) can receive the user's mobile aircraft boarding pass through a pre-check-in process prior to step (S270) and store the user's mobile aircraft boarding pass data.

[0355] In the above step (S250), the GPS coordinates of the terminal device (200) of the airport bus are used as the GPS coordinates of the airport bus.

[0356] The step (S250) of authenticating the user's intention to board an aircraft may include: a step of determining whether the airport bus has entered a preset designated section based on GPS data of the airport bus; a step of determining whether the user's GPS location matches the GPS location of the airport bus at at least one point on the preset designated section if the airport bus has entered the preset designated section; and a step of acknowledging the user's intention to board an aircraft if it is determined that the user's GPS location matches the GPS location of the airport bus at at least one point on the preset designated section.

[0357] In the above step (S250), the designated section may be a certain section including the airport regular route among the entire operation section of the airport bus. The designated section corresponds to a section of operation where arrival at the airport is guaranteed without the possibility of derailment. In some embodiments, the designated section may be part or all of the section between the first airport stop and the non-airport stop immediately before the first airport stop in a one-way operation section toward the airport. For example, if the airport bus is heading to Incheon Airport, the designated section may be part or all of the operation section of the airport bus on the Incheon Airport Expressway.

[0358] In some embodiments, the terminal device (200) of the airport bus may initiate an operation (S250) to authenticate the user's intention to board the aircraft when the GPS coordinates of the airport bus enter a designated section.

[0359] The terminal device (200) of the above airport bus compares the GPS coordinates of the airport bus and the user's terminal device 100 to confirm that they are in the same location within a certain time interval, and the user of the terminal device (100) within a certain distance can be authenticated as having the intention to board the aircraft (S250).

[0360] In the above step (S270), the baggage linkage code is a conditional temporary identification information assigned to the user's baggage before it is finally determined to be checked baggage. The baggage linkage code is a tracking code assigned by the system (1) to manage checked baggage from the trunk of the airport bus to the airport security inspection system (400). The baggage linkage code is an identification code different from the information on the baggage tag previously issued by an airline when checking in checked baggage, such as the baggage identification code (or airline baggage identification code). The baggage linkage code is assigned to the baggage of a user who uses the airport bus-linked checked baggage processing method of the system (1). Among the checked baggage loaded onto an aircraft, the checked baggage of a user who does not use the system (1) is not assigned a baggage linkage code. On the other hand, among the checked baggage loaded onto an aircraft, the checked baggage of a user who uses the system (1) is assigned the baggage linkage code of step (S270) in addition to the conventional baggage tag information (e.g., airline baggage code).

[0361] The above baggage linkage code consists of text and / or numbers, such as alphabets. Depending on the embodiments, the baggage linkage code may be transmitted to the user in digital form rather than printed on a baggage tag attached to the baggage, or may be transmitted to the user in digital form while simultaneously being printed on a label sticker via the employee's terminal (600).

[0362] If a user wishes to check in multiple pieces of luggage, each piece of luggage will be assigned a different baggage linking code.

[0363] In some embodiments, the terminal device (200) of the airport bus may generate a baggage linkage code by combining the identification information of the airport bus, the identification information of the bus stop where the user boarded, the user's identity information, and / or the baggage identification information assigned to the baggage. This allows the server (300) to identify the airport bus that delivered the baggage of the user assigned the baggage linkage code.

[0364] In the above step (S290), the server (300) can receive and store the user's identity verification result and the corresponding baggage linkage code confirmed in step (S230) from the terminal device (200) of the airport bus.

[0365] In addition, the method for linking checked baggage of the airport bus may include a step (S310) of transmitting a baggage linkage code of a user to a terminal device of the user based on the result of the user's identity verification in the server (300); a step (S320) of receiving, in the server (300), a baggage bag image photographed of the user's baggage bag and reading result data for an X-RAY image of the baggage bag from an airport security screening system (400) of an airport; and a step (S350) of linking the reading result data for the baggage bag with the baggage linkage code of the user based on a matching result of the baggage bag image of the airport security screening system (400) and the user's baggage bag image, and a matching result of the reading result data of the airport security screening system (400) and the user's item data. Additionally, in some embodiments, the method for linking checked baggage of the airport bus may further include a step (S311) of transmitting baggage tag information to the employee terminal (600) so that the employee terminal (600) prints baggage tag information including a baggage linking code and an airline baggage identification code.

[0366] In the above step (S310), the server (300) can transmit a linkage processing message including the corresponding baggage linkage code to the terminal device (100) of the user corresponding to the user's identity verification result confirmed in step (S230).

[0367] FIG. 11 is a schematic diagram of a baggage linkage processing message according to various embodiments of the present application.

[0368] Referring to FIG. 11, the linkage processing message may include a baggage linkage code. In some embodiments, the linkage processing message may further include information on items recognized as baggage and information on the baggage bag.

[0369] In the above step (S311), the server (300) transmits a baggage list for each airport bus, which lists the baggage loaded on the airport bus, to the employee terminal (600). The baggage list for each airport bus is a list consisting of user account information and baggage tag information. If multiple users board the airport bus, the baggage list for each airport bus is composed of baggage tag information for the baggage bags of multiple users.

[0370] The employee terminal (600) can print a label sticker containing baggage tag information from the baggage list via an internally installed print unit or an externally connected wired / wireless printer. The label sticker is attached to the user's checked baggage (S311). The label sticker may include a two-dimensional image containing baggage tag information. The two-dimensional image may be expressed as a barcode or QR code.

[0371] In various embodiments of the present application, the label sticker includes a baggage linkage code (e.g., a linkage number). Additionally, in some embodiments, the label sticker may further include flight information, passenger information, destination information, baggage date, and baggage identification information for airline identification. The airline baggage identification information may be expressed in the form of a barcode.

[0372] That is, the label sticker printed by the employee terminal device (600) in the above step (S311) may be in a form in which a baggage linkage code according to the present invention is added in addition to the flight information, passenger information, destination, date, and barcode that appear on a conventional baggage tag.

[0373]

[0374]

[0375] In various embodiments of the present application, the step (S311) of transmitting the baggage tag information to the employee terminal (600) may include transmitting a baggage bag image corresponding to the baggage linkage code in the baggage tag information. This allows the employee to know which baggage bag should be labeled with which baggage tag information, thereby preventing incorrect label sticker attachment.

[0376]

[0377] In the above step (S320), the luggage bag image may be an image of the exterior of the luggage bag captured using wavelengths in the visible light range. The luggage bag image of the above step (S320) may correspond to the luggage bag image of the step (S120).

[0378] In the above step (S320), the reading result data for the X-RAY image of the luggage bag includes the recognition result for at least one item contained in the user's luggage bag based on the X-RAY, whether the item corresponds to checked baggage, and the type of the checked baggage.

[0379] For example, the result of reading the X-RAY image of the luggage bag of FIG. 6 may include item information (e.g., item type) for each of at least some of the luggage of FIG. 6.

[0380] If a number of items correspond to restricted checked baggage, the server (300) can receive a list of restricted checked baggage as an identification result (S320).

[0381] In the above step (S350), the server (300) compares and analyzes the baggage image of the airport security inspection system (400) received in the above step (S320) and the user's baggage image received in the above step (S120) to determine whether the baggage shown in each bag image matches each other. The matching result can be calculated as a matching probability. To this end, the server (300) can analyze whether different bag images represent the same bag using various image analysis methods.

[0382] In the above step (S350), the server (300) can determine whether the item information recognized from the X-RAY image in the result of reading the baggage inside the baggage bag received from the airport security inspection system 300 matches the item information recognized from the baggage data generated in step (S130). The matching result can be calculated as a matching probability. In some embodiments, the server (300) can calculate the matching result as a matching probability between types of goods (e.g., product groups) (S350).

[0383] The above server (300) can search for the baggage linkage code of the user with the highest combination of the baggage matching result and the item matching result, and associate the searched baggage linkage code with the baggage shown in the X-RAY image and the reading result thereof (S350).

[0384] In addition, the method for connecting checked baggage of the airport bus may include a step (S360) of receiving a message regarding the result of the baggage inspection of the user from the airport security inspection system; and a step (S370) of transmitting a notification message to the user's terminal device (100) based on the message regarding the result of the baggage inspection of the user received from the airport security inspection system (400).

[0385] The inspection result message indicates whether the baggage can pass through airport security and be loaded onto the aircraft.

[0386] The above inspection result message may be a rejection message indicating that checked baggage with restricted baggage was found during baggage inspection, or a pass message indicating that checked baggage with restricted baggage was not found during baggage inspection.

[0387] In various embodiments of the present application, the step (S370) of transmitting the notification message may include: a step (S371) of transmitting a first notification message to the user's terminal device based on a baggage linkage code in the drop message when a drop message indicating that checked baggage has been searched in baggage inspection from the airport security screening system is received; and a step (S372) of transmitting a second notification message including a baggage linkage code to the user's terminal device when a pass message is received from the airport security screening system based on a baggage linkage code in the pass message when a pass message indicating that checked baggage has not been searched in baggage inspection from the airport security screening system is received.

[0388] In the above step (S370), the notification message is a baggage consignment notification message that informs the user whether or not the baggage has been finally consigned. The baggage consignment notification message may be provided as a first notification message or a second notification message depending on the security screening results.

[0389] As a result of the above security check, the drop message may include the baggage linking code, and the first notification message may include the baggage check location and the baggage linking code.

[0390] The above-mentioned "disqualified" message in the security screening results indicates that the user's baggage failed the security screening due to a restricted baggage being detected among the user's baggage items. The airport security screening system (400) obtains the user's baggage linkage code based on the baggage tag attached to the user's baggage, and transmits a "disqualified" message including the baggage linkage code and the disqualified information as a result of the screening to the server (300).

[0391] The first notification message may include at least one piece of information from among the baggage check location and baggage tag information, for example, the baggage linkage code. The server (300) may search for a user's terminal device (100) corresponding to the baggage linkage code in the dropped message, and transmit a first notification message including the baggage check location and the baggage linkage code to the searched user's terminal device (100) (S371).

[0392] The first alarm message indicates the location of the baggage checkpoint. It also indicates that the baggage checkpoint has been rejected. Consequently, the first alarm message can summon a user assigned a specific baggage linkage code to the baggage checkpoint.

[0393] A "Pass" message in the security screening results indicates that the user's baggage has passed the security screening, as no checked baggage was detected during the baggage security screening. The airport security screening system (400) obtains the user's baggage linkage code based on the baggage tag attached to the user's baggage, and transmits a "Pass" message including the baggage linkage code and the pass information as the screening result to the server (300).

[0394] The second notification message above indicates baggage tag information and the fact that the user has passed the security check. The second notification message can notify the user of the fact that the user has passed the security check (S372).

[0395] In some embodiments, the step (S372) may be to transmit a second notification message including a baggage management number issued by the airport security screening system to the user's terminal device. Here, the baggage management number is a number printed on a baggage tag by the security screening system (400) and attached to a baggage bag. Generally, the baggage management number is issued and attached when checking in a baggage bag at a check-in desk in the airport. The baggage management number serves as a baggage check-in confirmation. -

[0396]

[0397] These airport bus baggage check-in systems and methods minimize the burden passengers face from completing check-in and baggage check-in procedures before boarding time. Ultimately, passengers can enjoy a quick and convenient flight.

[0398] Additionally, passengers can save energy by easily using public transportation such as airport buses instead of their personal vehicles, and secure public transportation's competitiveness over other means of transportation to reach the airport.

[0399] Meanwhile, airports have check-in desk spaces, including passenger waiting areas, security areas, CIQ (Customs, Immigration, Quarantine) areas, duty-free shops, restaurants, rest areas, boarding waiting areas, and other internal airport spaces. If a user skips the check-in desk visit through the checked baggage linking system (1), the check-in desk space can ultimately be replaced with other internal airport spaces. This allows airport operators and designers to utilize internal space more efficiently for aircraft boarding.

[0400]

[0401] When implementing an embodiment of the present invention using hardware, ASICs (application specific integrated circuits) or DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), etc. configured to perform the present invention may be provided in the processor of the present invention.

[0402] Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented on a general-purpose digital computer that runs the program using a computer-readable medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable storage medium through various means. It should be understood that the program storage devices that can be used to describe a storage device including executable computer code for performing various methods of the present invention do not include transient objects such as carrier waves or signals. The computer-readable storage medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0403] The embodiments described above are combinations of components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented without being combined with other components or features. Furthermore, it is also possible to form an embodiment of the present invention by combining some components and / or features. The order of operations described in the embodiments of the invention may be changed. Some components or features of one embodiment may be included in another embodiment or may be replaced with corresponding components or features of another embodiment. It is self-evident that claims that are not explicitly cited in the scope of the patent may be combined to form an embodiment or may be incorporated as a new claim through a post-application amendment.

[0404] It will be apparent to those skilled in the art that the present invention can be embodied in other forms without departing from the technical spirit and essential characteristics of the present invention. Therefore, the above embodiments should be considered illustrative rather than restrictive in all respects. The scope of the present invention should be determined by a reasonable interpretation of the appended claims and all possible variations within the scope of equivalents of the present invention.

[0405]

[0406] The system and method for linking checked baggage of an airport bus using AI-based recognition and GPS coordinates according to the embodiments of the present application can provide various advantages in the field of airport bus / baggage handling, and thus have industrial applicability in the field.

Claims

1. In the method of connecting checked baggage on an airport bus, A step of obtaining mobile airport bus boarding pass data for the airport bus that the user will use to arrive at the airport to board the aircraft, through the terminal device of the user who will board the aircraft with baggage; A step of analyzing a baggage image of a user's baggage to be loaded onto the aircraft, which is photographed by the user's terminal device, and an image of the user's baggage showing the inside of the user's baggage, to produce the user's baggage data, and transmitting the user's baggage data to a server; A step of analyzing a facial image of a user boarding an airport bus, the facial image of the user being photographed by the terminal device of the airport bus, to confirm the identity of the user; A step of authenticating the user's intention to board an aircraft based on the GPS data of the user and the GPS data of the airport bus received from the terminal device of the user whose identity has been verified by the terminal device of the airport bus; A step of requesting and obtaining mobile boarding pass data for an aircraft of a user whose intention to board the aircraft has been authenticated by a terminal device of the airport bus, and generating a baggage linkage code corresponding to the mobile boarding pass of the user based on the item data of the user who boarded the airport bus, and generating the baggage linkage code; - The user's terminal device stores the user's mobile boarding pass data for the aircraft in advance through a pre-check-in process; A step of transmitting the user's identity verification result and baggage linking code to the server by the terminal device of the airport bus; A step of transmitting a baggage linking code of the user to the terminal device of the user based on the result of the user's identity verification at the server; In the above server, the baggage bag image captured by the airport security inspection system of the airport and the reading result data for the X-RAY image of the baggage bag are received, and the reading result includes an identification result for at least one item contained in the baggage bag of the user, A step of associating the reading result data for the baggage bag with the baggage linkage code of the user based on the matching result of the baggage bag image of the airport security screening system and the baggage bag image of the user, and the matching result of the reading result data of the airport security screening system and the item data of the user; A step of transmitting a notification message to the user's terminal based on a message regarding the user's baggage inspection result received from the airport security inspection system; including; method.

2. In the first paragraph, the step of authenticating the user's intention to board the aircraft based on the user's GPS data and the airport bus' GPS data received from the user's terminal device whose identity has been verified; A step of determining whether the airport bus has entered a preset designated section based on GPS data of the airport bus, wherein the designated section is a certain section including the airport regular route among the entire operating sections of the airport bus; When the airport bus enters a preset designated section, a step of determining whether the GPS location of the user and the GPS location of the airport bus match at least one point on the designated section; and A step of recognizing the user's intention to board an aircraft when it is determined that the user's GPS location and the airport bus' GPS location match each other at at least one point on the designated section; characterized in that it includes; method.

3. In the first paragraph, the step of analyzing the image of the user's luggage showing the inside of the user's luggage bag and calculating the user's item data is as follows: A step of recognizing the user's luggage by applying the image of the user's luggage to a first deep learning-based recognition model; A step of searching for weight data of the user's luggage through the Internet or a server based on the recognition result of the luggage bag; A step of applying the user's luggage image to a deep learning-based segmentation model to segment the item area in which luggage appears in the user's luggage image by each class of luggage and extracting the segmented item area as a sub-image; A step of recognizing an item appearing in the user's luggage image by applying a sub-image of the extracted item area to a second deep learning-based recognition model; A step of searching for weight data of items in the user's luggage bag through the Internet or a server based on the recognition result of the items; and A step of calculating the weight information of the user's baggage based on the weight search result of the recognized baggage bag and the weight search result of the recognized item; comprising; method.

4. In the third paragraph, the step of calculating the user's product data is: If the class of the above recognized item corresponds to a pouch, a step of requesting input of information about the items contained within the pouch; and A method for retrieving weight data for a pouch based on information about an item inside an input pouch and size data of the pouch calculated from a result of segmenting an item area in which the pouch appears, characterized in that the method further includes: method.

5. In paragraph 3, The above deep learning-based segmentation model is learned using a first training data set, wherein the first training data set is composed of a plurality of training samples, and each training sample is composed of training data and label data. The training data includes at least one training image, a sample image of an incomplete shape representing a learning sample item having an incomplete shape, and a sample image of a complete shape representing an article having a complete shape as an article shown in the sample image of the incomplete shape, Label data represents the actual value of the item type of the luggage that appears in the training image. The learned deep learning-based segmentation model is characterized in that the luggage image is input. method.

6. In the third paragraph, the second deep learning-based recognition model includes an input layer that inputs the sub-image as an input image, an encoder including one or more encoding blocks, a decoder including one or more decoding blocks, and a classification layer that recognizes and outputs an item appearing in the sub-image based on an encoding vector output from the decoder. The above second deep learning-based recognition model is, Split the input sub-image into multiple patches - each patch has a fixed size, Convert each patch into a patch embedding vector, Combine the position embedding value representing the location of each patch with the patch embedding vector of that patch, The combined vector is input to the encoder and processed to produce an encoding vector, Recognize the item appearing in the sub-image based on the encoding vector, It is configured to output the recognition results, The above encoder outputs class tokens as the encoding vector through a multi-head attention mechanism, and is characterized in that it learns the relationship between multiple patches through the multi-head attention mechanism. method.

7. In paragraph 6, The second deep learning-based recognition model is characterized in that the foundation model pre-trained with a specific data set is fine-tuned to recognize items of imperfect or complete shape contained in a luggage bag. method.

8. In the third paragraph, the step of calculating the user's product data is: Further comprising a step of calculating size information of an item appearing in an item area based on the size information and segmentation result of the above luggage bag; The step of recognizing the items shown in the image of the user's luggage is as follows: Applying the sub-image in which the above luggage appears and the size information of the above item to the second deep learning-based recognition model, The above second deep learning-based recognition model is, Convert the size information of the normalized item into a size information tensor, generate tensor data of the baggage based on the feature map and size information tensor extracted from the sub-image, and input the generated tensor data of the baggage into a fully connected layer as an intermediate operation result. The second deep learning-based recognition model is characterized in that it recognizes an item appearing in a sub-image having an incomplete shape by considering the characteristics of the baggage having an incomplete shape through the above fully connected layer. method.

9. In the first paragraph, the step of transmitting the notification message is: When receiving a drop message indicating that checked baggage has been searched for restricted baggage in baggage inspection from the airport security screening system, a step of transmitting a first notification message to the user's terminal based on a baggage linkage code in the drop message - - the drop message includes the baggage linkage code, and the first notification message includes a baggage check location and the baggage linkage code -; and When receiving a passage message based on a baggage linkage code in a passage message from the airport security inspection system, a step of transmitting a second notification message including the baggage linkage code to the user's terminal device. characterized by including, method.

10. In paragraph 1, A step of detecting items corresponding to restricted checked baggage among the items recognized in the item data based on restricted checked baggage information pre-stored in the user's terminal device before the user boards the airport bus; and Including a step of providing a user with a warning message including the detection result of the checked baggage when an item corresponding to a restricted checked baggage is included among the recognized items; method.

11. A computer-readable recording medium recording a program for performing a method for connecting checked baggage of an airport bus according to any one of claims 1 to 10.

12. In the airport bus checked baggage linkage system using AI-based recognition and GPS coordinates, A user's terminal device that acquires and stores mobile boarding pass data for an airport bus by performing a purchase process for a boarding pass for an airport bus that a user will use to arrive at an airport to board an aircraft, creates a baggage bag image by photographing the user's baggage to be loaded onto the aircraft and a baggage image by photographing the inside of the user's baggage, analyzes the baggage image to calculate the user's item data, transmits the baggage bag image by photographing the user's baggage to be loaded onto the aircraft, the user's baggage image, and the user's baggage data to a server, and performs a pre-check-in process for the aircraft to acquire and store mobile boarding pass data for the aircraft. An airport bus terminal device that calculates GPS data of an airport bus, takes a picture of the face of a user boarding the airport bus to generate facial image data of the user, analyzes the facial image of the user to calculate identity data of the user, authenticates the user's intention to board an airplane based on the GPS data of the user received from the terminal device of the identified user and the GPS data of the airport bus, requests and obtains the authenticated user's mobile airplane boarding pass data from the terminal device of the user, generates a baggage linkage code corresponding to the user's mobile boarding pass based on the item data of the user boarding the airport bus, and transmits the baggage linkage data including the baggage linkage code to the terminal device of the user through a server, and A server characterized in that it comprises a server for receiving a baggage bag image photographed from an airport security screening system installed at an airport and reading result data for an X-RAY image of the baggage bag, wherein the reading result includes an identification result for at least one item contained in the user's baggage bag, receiving the user's baggage bag image and the user's item data from the user's terminal device, associating the reading result data for the baggage bag with the user's baggage linkage code based on a matching result of the baggage bag image of the airport security screening system and the user's baggage bag image and a matching result of the reading result data of the airport security screening system and the user's item data, and transmitting a notification message to the user's terminal device based on a message about the user's baggage screening result received from the airport security screening system. System.

13. In paragraph 12, Further comprising a staff terminal configured to print baggage tag information including the above baggage linking code as a label sticker. System.

Images