A machine vision-based departure baggage control system

Abstract

This invention discloses a machine vision-based departure baggage check-in control system, which can optimize and simplify the baggage check-in process, improve check-in efficiency, optimize airport operations, and enhance passenger experience. The technical solution includes: a user interface module for providing airport staff with an operation and monitoring interface; a vision acquisition module for real-time monitoring of the baggage check-in status, capturing images and videos of the baggage during transportation; a vision processing module for collecting and processing data from the vision acquisition module and the control module, identifying the baggage status through image processing and algorithm analysis, and tracking any anomalies that may occur during transportation; and a control module for controlling the operation and shutdown of the corresponding belt conveyor based on the output of the vision processing module.

Description

Technical Field

[0001] This invention relates to the field of airport terminal baggage handling technology, specifically to a departure baggage handling control system based on machine vision. Background Technology

[0002] Figure 1 This illustrates a common airport departure baggage handling system layout, such as... Figure 1 As shown, the departure baggage system of existing small and medium-sized airports generally consists of a check-in conveyor subsystem, belt conveyors, turning conveyors, and flat baggage carousels (inclined baggage carousels). Baggage is first weighed and its length measured on the electronic weighing conveyor (DD7) of the check-in conveyor subsystem, then sent to the X-ray security screening conveyor (ZD7) for security inspection. After passing security, it is sent to the injection conveyor (GD7) to queue for entry into the collection conveyor (PD5) at an empty window. The collected baggage then passes through a series of belt conveyors and a turning input machine (ZW2D) before being sent to the sorting carousel (ZP5) for sorting and delivery to the corresponding flight.

[0003] The control process of the existing departure baggage system is as follows.

[0004] When departing passengers check in at the check-in counters in the departure hall, they place their luggage on the weighing conveyor. The check-in staff will weigh, check the dimensions, and affix tags to the luggage, which will then be assigned a flight number. If the luggage exceeds the size or weight limits, the passenger must hand it over to airport security personnel for oversized / overweight baggage handling.

[0005] After being labeled, standard baggage is scanned by a barcode scanner and conveyed from the electronic weighing conveyor to the dual-channel X-ray security screening conveyor. The two conveyors are arranged in parallel, sharing one X-ray security screening conveyor. Since each X-ray conveyor can only scan baggage on one conveyor at a time, no baggage can be on either conveyor simultaneously. In automatic operation mode, the system ensures that only one of the two parallel X-ray security screening conveyors has baggage on it at any given time.

[0006] After being inspected by the security conveyor, baggage arrives at the end of the loading conveyor to wait. Only when a safe clearance signal is received can baggage be sent to the collection conveyor; if an unsafe signal is received, the loading conveyor is locked, awaiting manual inspection or bag opening by security personnel; the check-in conveyor line can only restart after the lock is released.

[0007] The baggage handling system employs an equal-probability injection window control technology to prevent baggage accumulation. Only one piece of baggage can be assigned to each window, ensuring an equal opportunity for baggage from each check-in counter to be sent to the baggage handling system.

[0008] Baggage arriving at the collection conveyor is automatically transported to the baggage sorting turntable via a series of conveying equipment such as belt conveyors and turning conveyors. The baggage is then manually loaded into the corresponding baggage trolleys or baggage containers according to the flight number.

[0009] The existing departure baggage system has the following drawbacks.

[0010] 1. Existing equipment uses photoelectric sensors to determine the status of bags that are blocked or jammed. However, when there is no photoelectric sensor at the location of the bag, the system cannot make a judgment. Setting up a centralized baggage handling system will cause the system to make false judgments.

[0011] 2. When a piece of luggage gets stuck inside the security scanner, the system cannot identify it, which may lead to the loss of a suspicious piece of luggage, causing the system to make a misjudgment, and in some cases, even missing a passenger's luggage in the security scanner.

[0012] 3. Lost baggage issues. Accurate baggage tracking, especially the retrieval of lost baggage, reduces the cost of handling irregular baggage. Prevents baggage from failing to be sorted and loaded onto the aircraft or sent to the wrong flight at any stage of transportation.

[0013] 4. After baggage is checked at the check-in counter and arrives at the departure carousel, the baggage handlers need to check each bag to avoid omissions, which is a labor-intensive task.

[0014] 5. Using RFID tags for baggage tracking is costly and uses non-renewable materials, which is not environmentally friendly and does not meet the requirements of sustainable development.

[0015] 6. During system use, belt misalignment is not easily detected, and by the time it is discovered, the belt is already severely worn. Summary of the Invention

[0016] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0017] The purpose of this invention is to solve the above-mentioned problems and provide a departure baggage check-in control system based on machine vision, which can optimize and simplify the baggage check-in process, improve check-in efficiency, optimize airport operations, and improve passenger experience.

[0018] The technical solution of this invention is as follows: This invention discloses a departure baggage check-in control system based on machine vision, the system comprising:

[0019] User interface module, used to provide airport staff with an operating and monitoring interface;

[0020] The visual acquisition module is used to monitor the status of baggage during the check-in process in real time and capture images and videos of the baggage during transportation.

[0021] The vision processing module is used to collect and process data from the vision acquisition module and the control module. Through image processing and algorithm analysis, it identifies the status of the luggage and tracks any abnormalities that may occur during the transportation process in real time.

[0022] The control module is used to control the operation and stop of the corresponding belt conveyor based on the output of the vision processing module.

[0023] According to an embodiment of the departure baggage check-in control system based on machine vision of the present invention, the vision processing module processes the information collected by the vision acquisition module to obtain information including baggage position and swing arm position; on the other hand, the vision processing module receives the signal fed back by the control module and sends instructions to the control module after integrating the results of vision processing.

[0024] According to one embodiment of the machine vision-based departure baggage control system of the present invention, the system further includes a storage module for storing historical data of the vision processing module, the historical data being used for learning and optimizing system performance.

[0025] According to an embodiment of the machine vision-based departure baggage check-in control system of the present invention, the specific control method of the control module includes:

[0026] Bag blockage situation: The vision processing module determines that the suitcase is stationary and sends a feedback signal to the control module. At the same time, the control module receives a signal that the belt conveyor is running and issues a bag blockage alarm signal.

[0027] In case of excessive length: The vision processing module first identifies whether there is luggage on the electronic scale conveyor. If there is luggage, it identifies the outline of the luggage, calculates the size of the luggage, and if it meets the specifications, it proceeds to the next step of the process normally. If it does not meet the specifications, it will trigger an alarm for oversized luggage.

[0028] Standby mode: The vision processing module determines whether there is luggage on the belt conveyor. If no luggage passes by within the set time period, the control module controls the belt conveyor to stop running and enters standby mode.

[0029] Carousel full load status: The vision processing module determines that the spacing between suitcases on the luggage carousel is less than the set length, and sends a feedback signal to the control module, indicating that the carousel is full.

[0030] In the case of luggage being left behind in the security screening machine: the vision processing module detects that the luggage has entered the security screening machine, but the luggage has not come out of the security screening machine after the set time limit. The control module determines that the luggage has been left behind in the security screening machine and issues an alarm.

[0031] Belt misalignment: When the vision processing module determines that the distance between the belt and the two side guard plates is inconsistent and the belt is closer to one side, it will remind the staff to perform maintenance. When the vision processing module determines that the belt edge is severely worn, it will remind the staff to perform maintenance.

[0032] According to an embodiment of the departure baggage check-in control system based on machine vision of the present invention, the system is further configured to operate the check-in process as follows:

[0033] Step 1-1: After the luggage is placed on the electronic scale, the departure baggage control system processes the images captured by the camera monitoring the electronic scale conveyor to determine whether the luggage is oversized. If the luggage is not oversized, the electronic scale is not overweight, and the camera has captured that there is luggage on the electronic scale, the system sends a signal to start the electronic scale conveyor. The conveyor stops when the luggage reaches the level of the check-in staff. If the luggage is oversized, overweight, or not for the flight, the counter will remind the check-in staff and passengers to take the luggage.

[0034] Steps 1-2: Baggage is tagged and check-in information is sent to the departure baggage control system when the check-in staff checks in the baggage.

[0035] Steps 1-3: Upon receiving check-in information and determining that the baggage has been tagged, the departure baggage control system sequentially sends start signals to the electronic scale conveyor, security screening machine conveyor, and baggage filling conveyor. Based on the images captured by the cameras monitoring the electronic scale conveyor and the baggage filling conveyor, the system determines that the electronic scale conveyor stops when the baggage leaves the scale. Based on the images captured by the cameras monitoring the baggage filling conveyor, the system determines that the security screening machine conveyor and baggage filling conveyor stop when the baggage reaches the rear of the baggage filling conveyor, awaiting the image interpretation results from the security screening machine.

[0036] Steps 1-4: The security screening machine provides the image analysis results. The departing baggage control system receives baggage on the collection conveyor according to the images collected by the camera on the monitoring belt conveyor and the window control technology. For baggage entering the conveyor, a window is reserved first. When the baggage reaches the position of the entry conveyor, the entry conveyor automatically puts the baggage into the collection conveyor. The entry conveyor is not allowed to put baggage into the collection conveyor at will.

[0037] According to an embodiment of the departure baggage control system based on machine vision of the present invention, the system is further configured to run the baggage tracking process as follows:

[0038] Step 2-1: Camera calibration: Calibrate each camera and obtain its intrinsic and extrinsic parameters;

[0039] Step 2-2: Image preprocessing and feature extraction: Image preprocessing refers to performing preprocessing on the images captured by each camera, including denoising, grayscale conversion, and binarization. Feature extraction refers to using feature extraction algorithms to extract points or regions with obvious features from the image.

[0040] Steps 2-3: Image matching and target tracking: Image matching refers to finding similar feature points between images captured by different cameras using a feature point matching algorithm; target tracking refers to continuously tracking a target based on the matched feature points using a target tracking algorithm.

[0041] Steps 2-4: Multi-camera data fusion and tracking optimization: Data fusion refers to combining tracking results from different cameras, and tracking optimization refers to using optimization algorithms to further optimize the tracking results;

[0042] Steps 2-5: The departure baggage control system transforms the position of objects from the camera's perspective to the global coordinate system, continuously tracks objects in each camera, generates the object's motion trajectory, associates the object trajectories in different cameras to form cross-camera object motion trajectories, and uses the position information in the global coordinate system to realize the trajectory association and tracking of baggage by matching the baggage's feature information.

[0043] According to an embodiment of the departure baggage check-in control system based on machine vision of the present invention, the system is further configured to operate the baggage opening control process as follows:

[0044] Step 3-1: The departure baggage control system determines whether baggage has arrived on the conveyor belt based on images captured by the cameras monitoring the baggage compartments.

[0045] Step 3-2: When there is luggage on the belt conveyor, the system determines whether the luggage is qualified or suspicious based on the image information recorded for the luggage.

[0046] Step 3-3: If the security screening machine at the check-in lane identifies the baggage as suspicious, determine if the swing arm of the flow divider is in the extended position.

[0047] Steps 3-4: If the security screening machine at the check-in lane determines that the baggage is standard, check if the swing arm of the flow divider is in the return position.

[0048] According to an embodiment of the departure baggage check-in control system based on machine vision of the present invention, the system is further configured with the following conveyor control flow:

[0049] Step 4-1: Conveyor Start-up: After the operator presses the start button, the system uses images captured by the camera to determine if there is any luggage on the conveyor belt, and then starts the system.

[0050] Step 4-2: Stopping the conveyor: After the operator presses the stop button, the system uses images captured by the camera to determine if there is any luggage on the conveyor belt, and then stops operation.

[0051] Step 4-3: Energy Saving Processing: The vision processing module analyzes the images captured by the camera in the vision acquisition module, determines whether there is luggage on each section of the conveyor belt and records the time. When the vision processing module detects that no luggage passes through the conveyor line within the set time, it sends an energy saving signal to the control module. The control module controls the belt conveyors to stop running sequentially, and the entire system enters the energy saving standby mode. In the energy saving standby mode, the vision processing module analyzes the images captured by the camera in the injection machine area of ​​the vision acquisition module, determines whether there is luggage on the injection conveyor, and sends an energy saving standby mode termination signal to the control module. The control module controls the belt conveyors to start sequentially.

[0052] Step 4-4: Bag Blockage Handling: When the conveyor is running, the system uses images captured by the camera to determine if a piece of luggage on the conveyor belt has not moved for a long time. If so, the system determines that a bag blockage has occurred, stops the conveyor, and issues an alarm. After clearing the blockage, pressing the reset button on the electrical control cabinet restores system operation. When the sorting turntable is running, the system uses images captured by the camera monitoring the turntable to determine if a bag blockage has occurred when the turntable is full of luggage. The system then alerts the handling workers to sort the luggage as soon as possible.

[0053] Steps 4-5: Operational Faults and Handling: When the conveyor's output signal and feedback signal are inconsistent, an operational fault occurs. When the entire system is in operation mode, the vision processing module analyzes the images captured by the monitoring belt conveyor camera to determine whether the belt conveyor is running. The module then compares the judgment result with the signal fed back by the control module. If they are inconsistent, the vision processing module issues a fault alarm, prompting manual handling.

[0054] Steps 4-6: Queue Management: When the downstream conveyor stops, the system uses images captured by the camera to determine if luggage is passing by the upstream conveyor. The system then controls the upstream conveyor to stop and wait. Once the downstream conveyor resumes operation, the upstream conveyor continues running. When the entire system is in operation mode, the control module receives a stop signal from the belt conveyor. The vision processing module analyzes the images captured by the camera on the previous belt conveyor section to determine if luggage has passed by. If luggage passes by, the vision processing module sends a signal to the control module to stop the previous belt conveyor section.

[0055] Steps 4-7: Emergency Stop Reset and Startup: After the emergency stop switch is pulled up, the conveyor should not start immediately. Only after pressing the reset button on the control cabinet will the conveyor start step by step.

[0056] Compared with existing technologies, this invention offers the following advantages: The machine vision-based departure baggage control system of this invention includes a user interface module, a vision acquisition module, a vision processing module, and a control module. First, the system of this invention removes components such as photoelectric sensors, foot switches, barcode scanners, and limit switches from existing systems, reducing the possibility of safety accidents caused by component aging, wear, or malfunction, thereby optimizing system composition and improving system reliability. Second, by using the system of this invention to control the departure baggage check-in process, check-in staff no longer need to step on the foot pedal; after attaching the baggage tag, they can wait for the system to automatically transport the baggage to the sorting carousel. The workers handling the baggage on the sorting carousel no longer need to verify baggage information; the system automatically verifies it and alarms when an anomaly is detected, thus simplifying the workflow. Third, the system of this invention enables full-process tracking of checked baggage, reducing workload and improving system efficiency. Fourth, the system of this invention can promptly detect changes in equipment, enabling pre-maintenance and pre-servicing, thereby extending system lifespan and reducing the frequency of emergency repairs, thus reducing maintenance pressure. Attached Figure Description

[0057] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0058] Figure 1 The layout diagram of the existing airport departure baggage system is shown.

[0059] Figure 2 A schematic diagram of an embodiment of the machine vision-based departure baggage control system of the present invention is shown.

[0060] Figure 3 A schematic diagram of the layout of the centralized baggage unpacking system for departing passengers is shown.

[0061] Figure 4 It shows Figure 2 The diagram shows the check-in process of the system.

[0062] Figure 5 It shows Figure 2 The diagram shows the baggage tracking process of the system.

[0063] Figure 6 It shows Figure 2 The diagram shows the control flow of the system's opening room.

[0064] Figure 7 It shows Figure 2 The diagram shows the conveyor control flow of the system. Detailed Implementation

[0065] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0066] Figure 2 The principle of an embodiment of the machine vision-based departure baggage control system of the present invention is illustrated. Please refer to [link / reference]. Figure 2 The system in this embodiment includes: a user interface module, a visual acquisition module, a visual processing module, and a control module. In addition, the system also includes a storage module. Combined with... Figure 3 The diagram shows the layout of the centralized baggage handling system for departing passengers. Figure 3 In this diagram, SX represents a camera, PD represents a belt conveyor, ZW represents a turning conveyor, ZP represents a sorting turntable, FLQ represents a swing arm diverter (vertical diverters and swing wheel diverters also apply), DD7 represents an electronic scale conveyor, ZD7 represents a security inspection machine conveyor, GD7 represents an injection conveyor, and D-1 and D-2 represent omnidirectional ball platforms. Cameras are categorized as follows: SX1, SX2, SX3, SX4 for monitoring electronic scale conveyors; SX5, SX10, SX11, SX15 for monitoring belt conveyors; SX6, SX7, SX8, SX9 for monitoring injection conveyors; SX12, SX13, SX14 for monitoring baggage handling areas; and SX16, SX17, SX18, SX19 for monitoring sorting turntables. The layout of each airport is different, and the arrangement of belt conveyors also varies. Figure 3 For matching purposes only Figure 2 This is an example of a system composition for illustrative purposes only.

[0067] The user interface module provides an operating and monitoring interface for airport staff to display baggage status, abnormal alarm information, etc.

[0068] The visual acquisition module is used to monitor the status of baggage during the check-in process in real time, capturing images and videos of the baggage during transportation. The visual acquisition module includes... Figure 3 The system shown in this embodiment uses hardware devices such as cameras. Data acquisition in this embodiment does not require sensors found in traditional systems, such as photoelectric sensors or limit sensors.

[0069] The vision processing module collects and processes data from the vision acquisition module and the control module. Through image processing and algorithm analysis, it identifies the status of the luggage and tracks any anomalies that may occur during transportation in real time. On one hand, the vision processing module processes the information collected by the vision acquisition module to obtain information such as the luggage position and swing arm position. On the other hand, the vision processing module receives signals from the control module, integrates the results of vision processing, and sends instructions to the control module.

[0070] In addition, the storage module also stores historical data from the vision processing module for learning and optimizing system performance. The hardware and software carriers of the vision processing module are the vision processing software and host computer system installed on the server, while the hardware devices of the storage module are storage hard drives installed on the server.

[0071] Algorithms for recognizing baggage status in the vision processing module include, for example, the Faster-R-CNN algorithm, the YOLO series algorithms, and the Mask-R-CNN algorithm. Algorithms for real-time baggage tracking in the vision processing module include, for example, deep learning-based target tracking algorithms, correlation filtering-based target tracking algorithms, optical flow-based target tracking algorithms, and Siamese network (Siamese network) tracking algorithms.

[0072] The control module is used to control the operation and stop of the corresponding belt conveyor based on the output of the vision processing module. The hardware of the control module consists of a PLC controller and related input / output modules, located in the electrical control box.

[0073] The specific control method of the control module is as follows.

[0074] Bag blockage situation: The vision processing module determines that the suitcase is stationary and sends a feedback signal to the control module. At the same time, the control module receives a signal that the belt conveyor is running and issues a bag blockage alarm signal.

[0075] For oversized luggage: The vision processing module first identifies whether there is luggage on the electronic scale conveyor. If there is luggage, it identifies the luggage's outline and calculates the luggage's dimensions. If the luggage meets the specifications, it proceeds to the next step normally; otherwise, an alarm is triggered for oversized luggage.

[0076] Standby status: The vision processing module determines whether there is luggage on the belt conveyor. If no luggage passes by within a set time period (e.g., 10 minutes, time adjustable), the control module controls the belt conveyor to stop running and enter standby mode.

[0077] Carousel full load status: The vision processing module determines that the spacing between suitcases on the luggage carousel is less than the set length (e.g., 1 meter), and sends a feedback signal to the control module that the carousel is full.

[0078] In the case of luggage being left behind in the security screening machine: The vision processing module recognizes that the luggage has entered the security screening machine, but the luggage has not come out of the security screening machine after a set time limit (e.g., 10 seconds). The control module determines that the luggage has been left behind in the security screening machine and issues an alarm.

[0079] Belt misalignment: When the vision processing module detects that the belt is unevenly positioned between its distance and the side guard plates, and is too close to one side (exceeding the initial limit), it alerts the staff that maintenance is required. The vision processing module also detects severe wear on the belt edges and alerts the staff that maintenance is necessary.

[0080] The machine vision-based departure baggage control system in this embodiment is configured to allow the following workflows: check-in process, baggage tracking process, baggage opening control process, and conveyor control process.

[0081] Combination Figure 3 As shown below, the above process will be explained in more detail.

[0082] Figure 4 The system's check-in process is shown and detailed below.

[0083] Step 1-1: After the luggage is placed on the electronic scale DD7, the departure baggage control system processes the image captured by the camera SX1 monitoring the electronic scale conveyor to determine if the luggage exceeds the length limit. If the luggage is not over the length limit, the electronic scale is not over the weight limit, and the camera SX1 detects luggage on the electronic scale, the system signals the electronic scale conveyor to start, stopping when the luggage reaches level with the check-in staff. For luggage that is over the length limit, over the weight limit, or not for the flight, the counter staff and passengers are notified to collect the luggage.

[0084] Steps 1-2: When the check-in staff checks in the baggage, baggage tags are affixed and the check-in information (flight information, passenger information, baggage information, etc.) is sent to the departure baggage control system.

[0085] Steps 1-3: Upon receiving check-in information and determining that the baggage has been tagged, the departure baggage control system sequentially sends start signals to the electronic scale conveyor, security screening machine conveyor, and baggage sorting conveyor. Based on images captured by camera SX1 monitoring the electronic scale conveyor and camera SX6 monitoring the baggage sorting conveyor, the system determines that the electronic scale conveyor stops when the baggage leaves the scale. Based on images captured by camera SX6 monitoring the baggage sorting conveyor, the system determines that both the security screening machine conveyor and the baggage sorting conveyor stop when the baggage reaches the rear of the sorting conveyor. The system then awaits the security screening machine's image processing results.

[0086] Steps 1-4: The security screening machine provides the image analysis results. Based on the images captured by cameras SX5 and SX11 on the monitoring belt conveyor, the departing baggage control system receives baggage on the collection conveyor PD5-0 using window control technology. Baggage entering the conveyor is first reserved at a designated window. When the baggage reaches the designated window, the conveyor automatically places it into the collection conveyor. The conveyor is not allowed to randomly place baggage into the collection conveyor. Window control technology ensures that baggage enters the collection conveyor with "equal probability and equal intervals."

[0087] Figure 5 The baggage tracking process of the system is shown below in detail.

[0088] Step 2-1: Camera Calibration. Each camera is calibrated to obtain its intrinsic parameters (such as focal length and optical center) and extrinsic parameters (such as the camera's position and orientation in three-dimensional space). The calibration process typically involves photographing a calibration board of known dimensions and extracting feature points from the board to calculate the camera's parameters.

[0089] After calculating the parameters, it is best to optimize them: by iteratively optimizing the algorithm, the accuracy of the camera parameters can be further improved, and the consistency of parameters between cameras can be ensured, so as to reduce errors in subsequent image matching and fusion processes.

[0090] Step 2-2: Image preprocessing and feature extraction.

[0091] Image preprocessing refers to the preprocessing of images captured by each camera, including steps such as noise reduction, grayscale conversion, and binarization. The preprocessing process aims to improve image quality and reduce the computational load and errors in subsequent processing.

[0092] Feature extraction refers to the process of using feature extraction algorithms (such as SIFT, SURF, ORB, etc.) to extract points or regions with distinctive features from an image. These feature points or regions will play a crucial role in subsequent image matching.

[0093] Steps 2-3: Image matching and target tracking.

[0094] Image matching refers to finding similar feature points between images captured by different cameras using feature point matching algorithms (such as FLANN, BFMatcher, etc.). The matching process needs to take into account factors such as the position, orientation, and scale of the feature points to ensure the accuracy of the matching.

[0095] Target tracking refers to the continuous tracking of a target using target tracking algorithms (such as Kalman filtering, particle filtering, DeepSORT, etc.) based on matched feature points. The tracking algorithm needs to continuously update information such as the target's position and velocity to cope with dynamic changes in the target.

[0096] Steps 2-4: Multi-camera data fusion and tracking optimization.

[0097] Data fusion refers to combining tracking results from different cameras to obtain more accurate and complete tracking information. The fusion process needs to take into account factors such as the positional relationship between cameras and overlapping areas to ensure the accuracy and consistency of the data.

[0098] Tracking optimization refers to using optimization algorithms (such as genetic algorithms, particle swarm optimization, etc.) to further optimize the tracking results. The optimization process aims to reduce tracking errors and improve the stability and accuracy of tracking.

[0099] Steps 2-5: The departing baggage control system transforms the position of objects from the camera's viewpoint to the global coordinate system, continuously tracks objects in each camera, generates their motion trajectories, and correlates the trajectories from different cameras to form cross-camera object motion trajectories. By matching baggage feature information (such as color, shape, texture, flight information, passenger information, security check image information, etc.), the system utilizes the position information in the global coordinate system to achieve baggage trajectory correlation and tracking.

[0100] Figure 6 The system's control flow for opening the packaging room is shown and detailed below.

[0101] Step 3-1: The departure baggage check-in control system determines whether baggage has arrived on the belt conveyor PD5-1 based on the images collected by the cameras SX12 and SX14 monitoring the baggage check-in area.

[0102] a) The vision processing module analyzes the images captured by the vision acquisition module (i.e., the camera monitoring the baggage compartment) in real time to determine whether any luggage has arrived on the belt conveyor PD5-1. If there is no luggage, no action is taken.

[0103] b) When the vision processing module determines that there is luggage, it determines the status of the luggage based on the luggage image information fed back by the control module;

[0104] c) The vision processing module determines that the luggage is normal luggage and checks the position of the swing arm of the FLQ diverter. When the swing arm is in the retracted position, neither the vision processing module nor the control module takes any action. When the swing arm is in the extended position, the vision processing module sends a command to the control module. The control module first stops the belt conveyor PD5-1, then controls the swing arm to retract, and restarts the belt conveyor PD5-1 after the swing arm retracts.

[0105] d) The vision processing module determines that the baggage is suspicious and checks the position of the swing arm of the FLQ diverter. When the swing arm is in the extended position, neither the vision processing module nor the control module takes any action. When the swing arm is in the retracted position, the vision processing module sends a command to the control module. The control module first stops the belt conveyor PD5-1, then controls the swing arm to extend, and restarts the belt conveyor PD5-1 after the swing arm extends.

[0106] e) Suspicious baggage passes through belt conveyor PD5-1 and belt conveyor PD5-2 in sequence to reach the baggage on the omnidirectional ball platform D-1 to wait for on-site inspection.

[0107] Step 3-2: When there is luggage on the belt conveyor PD5-1, the luggage is judged as qualified luggage / suspicious luggage based on the judgment information of the luggage recorded by the system.

[0108] Step 3-3: If the security screening machine at the check-in lane identifies the baggage as suspicious, determine whether the swing arm of the flow divider FLQ is in the swing-out position.

[0109] a) When the swing arm of the diverter FLQ is in the out position, the belt conveyor PD5-1 continues to run. The luggage passes through the diverter FLQ and the conveyor belt PD5-2 and arrives at the omnidirectional ball platform D-1.

[0110] (b) When the swing arm of the flow divider FLQ is in the return position, the system checks whether there is luggage on the belt conveyor PD5-4 within the swing arm's range. If there is no luggage, the system controls the swing arm of the flow divider FLQ to extend, allowing the luggage to pass through the flow divider FLQ and conveyor belt PD5-2 to reach the omnidirectional ball platform D-1. If there is luggage, the belt conveyor PD5-1 stops running and waits until there is no luggage on conveyor belt PD5-4. Then, the system controls the swing arm of the flow divider FLQ to extend, and the belt conveyor PD5-1 starts running, allowing the luggage to pass through the flow divider FLQ and conveyor belt PD5-2 to reach the omnidirectional ball platform D-1.

[0111] c) Luggage on the omnidirectional ball platform D-1 awaits on-site inspection.

[0112] d) Re-inspection personnel place the manually inspected and approved baggage onto the security screening machine conveyor ZD7. The departing baggage check-in control system determines whether there is baggage on the security screening machine conveyor ZD7 based on the image captured by the camera SX13 monitoring the baggage opening area. If the result indicates the presence of baggage, the system controls the operation of security screening machine conveyor ZD7 and belt conveyor PD5-3. Based on the image captured by the camera SX14 monitoring the baggage opening area, the system determines when the baggage has reached the rear end of belt conveyor PD5-3, and then controls security screening machine conveyor ZD7 and belt conveyor PD5-3 to stop. The system then awaits the image analysis result from the security screening machine in the baggage opening area.

[0113] e) If the image analysis result of the security scanner in the private room is still suspicious, the system will prompt the on-site re-inspection personnel to remove the luggage for re-inspection. If the image analysis result is normal, the system will determine, based on the images captured by the camera SX14 monitoring the private room and the camera SX15 monitoring the belt conveyor, that there is no bag collision, and will control the belt conveyor PD5-3 to run, and put the luggage into the belt conveyor PD5-4.

[0114] Steps 3-4: If the security screening machine at the check-in lane determines that the baggage is standard, check whether the swing arm of the flow divider FLQ is in the return position.

[0115] a) When the swing arm of the diverter FLQ is in the swing-back position, the belt conveyor PD5-1 continues to run, and the luggage is sent to the sorting turntable via conveyor belts PD5-4 and PD5-5 to wait for manual sorting.

[0116] (b) The FLQ splitter arm is in the extended position. The departure baggage control system uses images captured by camera SX14 (monitoring the baggage handling area) and camera SX15 (monitoring the belt conveyor) to check if there is baggage on conveyor PD5-4 within the splitter arm's swing arm range. If there is no baggage, the system controls the FLQ splitter arm to swing back. If there is baggage, conveyor PD5-1 stops running and waits until there is no baggage on conveyor PD5-4 within the splitter arm's swing arm range before controlling the FLQ splitter arm to swing back, and conveyor PD5-1 resumes operation.

[0117] Figure 7 The conveyor control flow of the system operation is shown and detailed below.

[0118] Step 4-1: Start the conveyor.

[0119] After the staff presses the start button, the system uses images captured by the camera to determine if there is any luggage on the conveyor belt, and then it can start. The starting sequence should be from downstream to upstream. The downstream conveyor starts first, and after the downstream conveyor starts, the upstream conveyor starts after a delay, which can be set.

[0120] The staff presses the system start button in the control module, which then sends a command to the vision processing module. The vision processing module analyzes the images captured by the cameras in the vision acquisition module to determine if there is luggage on each section of the conveyor belt. If there is no luggage on the entire conveyor line, the vision processing system sends a start signal to the control module, which then controls the conveyor lines to start sequentially from bottom to top at 1-second intervals. If there is luggage on the entire conveyor line, the vision processing system issues an alarm signal, prompting manual intervention.

[0121] Step 4-2: Stop the conveyor.

[0122] After the staff presses the stop button, the system uses images captured by the camera to determine that there is no luggage on the conveyor belt and can stop operation. The system should stop in the order from upstream to downstream. The upstream conveyor stops first, and after the upstream conveyor stops, the downstream conveyor stops after a delay, which can be set to ensure that the luggage is cleared.

[0123] The staff presses the system stop button in the control module, which then sends a command to the vision processing module. The vision processing module analyzes the images captured by the camera in the vision acquisition module to determine if there is luggage on each conveyor belt. If the vision processing module determines that there is no luggage on this conveyor belt or any of the upstream conveyor belts, it sends a stop signal to the control module, which then stops the conveyor belt after 1 second.

[0124] Step 4-3: Energy-saving treatment.

[0125] The system uses images captured by cameras to determine if there is any luggage on the conveyor belt and if a certain time has elapsed (adjustable as needed). The waiting time for the conveyor to enter energy-saving mode is an adjustable parameter available to the user, who can set it according to their usage. Furthermore, the control module employs effective measures to prevent interference from false signals and to prevent disruption of normal energy-saving functions. When the system needs to enter energy-saving mode, the conveyor stops after a delay from top to bottom; when the system is activated, the conveyor starts after a delay from bottom to top.

[0126] The vision processing module analyzes the images captured by the camera in the vision acquisition module to determine whether there is luggage on each section of the conveyor belt and records the time. When the vision processing module detects that no luggage has passed through the conveyor line for 20 minutes, it sends an energy-saving signal to the control module; the control module then controls the conveyor belts to stop sequentially from top to bottom at 1-second intervals, and the entire system enters energy-saving standby mode.

[0127] In energy-saving standby mode, the vision processing module analyzes the images captured by the camera in the injection machine area of ​​the vision acquisition module, determines that there is luggage on the injection conveyor, and sends an energy-saving standby mode termination signal to the control module; the control module controls the belt conveyor to start sequentially from top to bottom at 1-second intervals.

[0128] Step 4-4: Packet blocking.

[0129] When the conveyor is running, the system uses images captured by the camera to determine if a piece of luggage on the conveyor belt has been stationary for an extended period. If so, the system identifies a blockage fault, stops the conveyor, and issues an alarm. After clearing the blockage, pressing the reset button on the electrical control cabinet restores system operation.

[0130] When the sorting turntable is running, the system monitors images captured by cameras SX16, SX17, SX18, and SX19 on the turntable. If the turntable is full of luggage, a baggage blockage fault is detected, and the handling workers are reminded to sort the luggage as soon as possible.

[0131] Steps 4-5: Operational Faults and Support Handling.

[0132] When the conveyor's operating output signal and operating feedback signal are inconsistent, an operational fault occurs.

[0133] When the entire system is in operation mode, the vision processing module analyzes the images captured by the monitoring belt conveyor camera to determine whether the belt conveyor is running; and compares the judgment result with the signal fed back by the control module. If they are inconsistent, the vision processing module will issue a fault alarm and prompt manual handling.

[0134] Steps 4-6: Queue processing.

[0135] When the downstream conveyor stops running (due to baggage blockage, malfunction, maintenance, queuing, etc.), the system uses images captured by the camera to determine if there is luggage passing by the upstream conveyor. When this happens, the system controls the upstream conveyor to stop and wait. Once the downstream conveyor starts running, the upstream conveyor continues to run.

[0136] When the entire system is in operation mode, the control module receives a stop signal from the belt conveyor. The vision processing module analyzes the image captured by the camera of the previous belt conveyor to determine whether there is luggage passing by. If there is luggage, the vision processing module sends a signal to the control module to stop the operation of the previous belt conveyor.

[0137] Steps 4-7: Emergency stop reset and restart.

[0138] After the emergency stop switch is pulled up, the conveyor should not start immediately. The conveyor will only start step by step after the reset button on the control cabinet is pressed.

[0139] In addition, the system in this embodiment also implements the following functions.

[0140] 1. When the system determines that the conveyor belt is misaligned based on the collected images, it will remind airport maintenance personnel to maintain the corresponding conveyor belt in a timely manner.

[0141] 2. When the system determines, based on the collected images, that luggage is lost or misplaced before reaching the sorting carousel, it will alert airport staff to check promptly.

[0142] 3. The system automatically compares baggage information processed at the check-in counter with baggage information leaving the sorting carousel. Any discrepancies are promptly alerted to on-site staff to avoid disrupting passenger journeys.

[0143] 4. The system will save photos of luggage passing through each conveyor belt section, so that passengers can refer to them if problems arise later.

[0144] In summary, the system of the present invention has the following technical effects.

[0145] 1. Optimized and simplified baggage check-in process, improved check-in efficiency, and optimized airport operations:

[0146] Improved resource utilization: Increased check-in efficiency means that more passengers can be processed per unit of time, making fuller use of resources such as check-in counters and self-service check-in equipment, improving the utilization efficiency of airport facilities, and reducing resource idleness and waste.

[0147] Speeding up passenger flow: A fast check-in process allows passengers to move from the check-in area to the waiting area and boarding gate more quickly, which speeds up passenger flow within the airport, helps alleviate congestion, improves overall airport operational efficiency, and ensures smooth airport operation.

[0148] Enhancing collaborative operational capabilities: An efficient check-in process can better connect and collaborate with other airport processes (such as security checks and waiting areas) to form an efficient and smooth operating system, improve the overall operational efficiency and service quality of the airport, and reduce problems such as flight delays caused by poor coordination between different processes.

[0149] Enables end-to-end tracking of passenger baggage check-in: Track your baggage like you would a package – scan the code / enter the baggage number to view the real-time progress and estimated carousel arrival time, reducing the error rate of passenger baggage.

[0150] 2. Improve passenger experience:

[0151] Reduced waiting time: With improved check-in efficiency, the time passengers spend queuing to check in is significantly reduced, allowing them to quickly complete check-in and enter the waiting area, making the travel process smoother and easier, and reducing travel fatigue and anxiety.

[0152] Improving travel satisfaction: Fast and convenient check-in processes and real-time baggage location display create a positive impression on passengers, enhancing their overall travel satisfaction and contributing to a strong brand image, making passengers more willing to choose that airport.

[0153] Increased travel flexibility: Improved check-in efficiency gives passengers more time to shop, dine, or rest at the airport, enriching their experience. It also provides passengers with more time to deal with unforeseen circumstances (such as needing to reschedule their trips after flight delays), thus increasing travel flexibility.

[0154] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0155] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A machine vision-based departure baggage check-in control system, characterized in that, The system includes: User interface module, used to provide airport staff with an operating and monitoring interface; The visual acquisition module is used to monitor the status of baggage during the check-in process in real time and capture images and videos of the baggage during transportation. The vision processing module is used to collect and process data from the vision acquisition module and the control module. Through image processing and algorithm analysis, it identifies the status of the luggage and tracks any abnormalities that may occur during the transportation process in real time. The control module is used to control the operation and stop of the corresponding belt conveyor based on the output of the vision processing module; In the case of bag blockage: the vision processing module determines that the suitcase is stationary and sends a feedback signal to the control module. At the same time, the control module receives a signal that the belt conveyor is running and issues a bag blockage alarm signal. In case of excessive length: The vision processing module first identifies whether there is luggage on the electronic scale conveyor. If there is luggage, it identifies the outline of the luggage, calculates the size of the luggage, and if it meets the specifications, it proceeds to the next step of the process normally. If it does not meet the specifications, it will trigger an alarm for oversized luggage. Standby mode: The vision processing module determines whether there is luggage on the belt conveyor. If no luggage passes by within the set time period, the control module controls the belt conveyor to stop running and enters standby mode. Carousel full load status: The vision processing module determines that the spacing between suitcases on the luggage carousel is less than the set length, and sends a feedback signal to the control module, indicating that the carousel is full. In the case of luggage being left behind in the security screening machine: the vision processing module detects that the luggage has entered the security screening machine, but the luggage has not come out of the security screening machine after the set time limit. The control module determines that the luggage has been left behind in the security screening machine and issues an alarm. Belt misalignment: When the vision processing module determines that the distance between the belt and the two side guard plates is inconsistent and the belt is closer to one side, it will remind the staff to perform maintenance. When the vision processing module determines that the belt edge is severely worn, it will remind the staff to perform maintenance.

2. The machine vision-based departure baggage check-in control system according to claim 1, characterized in that, The vision processing module processes the information collected by the vision acquisition module to obtain information including the luggage position and the swing arm position. On the other hand, the vision processing module receives the signals fed back by the control module and sends instructions to the control module after synthesizing the vision processing results.

3. The machine vision-based departure baggage check-in control system according to claim 1, characterized in that, The system also includes a storage module for storing historical data from the vision processing module, which is used to learn and optimize system performance.

4. The machine vision-based departure baggage check-in control system according to claim 1, characterized in that, The system is also configured to run the check-in process as follows: Step 1-1: After the luggage is placed on the electronic scale, the departure baggage control system processes the images captured by the camera monitoring the electronic scale conveyor to determine whether the luggage is oversized. If the luggage is not oversized, the electronic scale is not overweight, and the camera has captured that there is luggage on the electronic scale, the system sends a signal to start the electronic scale conveyor. The conveyor stops when the luggage reaches the level of the check-in staff. If the luggage is oversized, overweight, or not for the flight, the counter will remind the check-in staff and passengers to take the luggage. Steps 1-2: Baggage is tagged and check-in information is sent to the departure baggage control system when the check-in staff checks in the baggage. Steps 1-3: Upon receiving check-in information and determining that the baggage has been tagged, the departure baggage control system sequentially sends start signals to the electronic scale conveyor, security screening machine conveyor, and baggage filling conveyor. Based on the images captured by the cameras monitoring the electronic scale conveyor and the baggage filling conveyor, the system determines that the electronic scale conveyor stops when the baggage leaves the scale. Based on the images captured by the cameras monitoring the baggage filling conveyor, the system determines that the security screening machine conveyor and baggage filling conveyor stop when the baggage reaches the rear of the baggage filling conveyor, awaiting the image interpretation results from the security screening machine. Steps 1-4: The security screening machine provides the image analysis results. The departing baggage control system receives baggage on the collection conveyor according to the images collected by the camera on the monitoring belt conveyor and the window control technology. For baggage entering the conveyor, a window is reserved first. When the baggage reaches the position of the entry conveyor, the entry conveyor automatically puts the baggage into the collection conveyor. The entry conveyor is not allowed to put baggage into the collection conveyor at will.

5. The machine vision-based departure baggage check-in control system according to claim 1, characterized in that, The system is also configured to run the baggage tracking process as follows: Step 2-1: Camera calibration: Calibrate each camera and obtain its intrinsic and extrinsic parameters; Step 2-2: Image preprocessing and feature extraction: Image preprocessing refers to performing preprocessing on the images captured by each camera, including denoising, grayscale conversion, and binarization. Feature extraction refers to using feature extraction algorithms to extract points or regions with obvious features from the image. Steps 2-3: Image matching and target tracking: Image matching refers to finding similar feature points between images captured by different cameras using a feature point matching algorithm; target tracking refers to continuously tracking a target based on the matched feature points using a target tracking algorithm. Steps 2-4: Multi-camera data fusion and tracking optimization: Data fusion refers to combining tracking results from different cameras, and tracking optimization refers to using optimization algorithms to further optimize the tracking results; Steps 2-5: The departure baggage control system transforms the position of objects from the camera's perspective to the global coordinate system, continuously tracks objects in each camera, generates the object's motion trajectory, associates the object trajectories in different cameras to form cross-camera object motion trajectories, and uses the position information in the global coordinate system to realize the trajectory association and tracking of baggage by matching the baggage's feature information.

6. The machine vision-based departure baggage check-in control system according to claim 1, characterized in that, The system is also configured to run the private room control process as follows: Step 3-1: The departure baggage control system determines whether baggage has arrived on the conveyor belt based on images captured by the cameras monitoring the baggage compartments. Step 3-2: When there is luggage on the belt conveyor, the system determines whether the luggage is qualified or suspicious based on the image information recorded for the luggage. Step 3-3: If the security screening machine at the check-in lane identifies the baggage as suspicious, determine if the swing arm of the flow divider is in the extended position. Steps 3-4: If the security screening machine at the check-in lane determines that the baggage is standard, check if the swing arm of the flow divider is in the return position.

7. The machine vision-based departure baggage check-in control system according to claim 1, characterized in that, The system is also configured to run the conveyor control process as follows: Step 4-1: Conveyor Start-up: After the operator presses the start button, the system uses images captured by the camera to determine if there is any luggage on the conveyor belt, and then starts the system. Step 4-2: Stopping the conveyor: After the operator presses the stop button, the system uses images captured by the camera to determine if there is any luggage on the conveyor belt, and then stops operation. Step 4-3: Energy-saving treatment; Step 4-4: Packet blockage handling; Steps 4-5: Troubleshooting and troubleshooting; Steps 4-6: Queue processing; Steps 4-7: Emergency Stop Reset and Startup: After the emergency stop switch is pulled up, the conveyor should not start immediately. Only after pressing the reset button on the control cabinet will the conveyor start step by step.

8. The machine vision-based departure baggage check-in control system according to claim 7, characterized in that, Step 4-3 further includes: the vision processing module analyzes the images captured by the camera of the vision acquisition module, determines whether there is luggage on each section of the conveyor belt and records the time. When the vision processing module detects that no luggage passes through the conveyor line within a set time, it sends an energy-saving signal to the control module. The control module controls the belt conveyors to stop running sequentially, and the entire system enters the energy-saving standby mode. In the energy-saving standby mode, the vision processing module analyzes the images captured by the camera of the injection machine area of ​​the vision acquisition module, determines that there is luggage on the injection conveyor, and sends an energy-saving standby mode termination signal to the control module. The control module controls the belt conveyors to start sequentially.

9. The machine vision-based departure baggage check-in control system according to claim 7, characterized in that, Step 4-4 further includes: when the conveyor is running, the system uses images captured by the camera to determine if a piece of luggage on the conveyor belt has not moved for a long time, and the system determines that a baggage blockage fault has occurred. The system then controls the conveyor to stop and sounds an alarm. After clearing the baggage blockage fault, the system resumes operation by pressing the reset button on the electrical control cabinet. When the sorting turntable is running, the system uses images captured by the camera monitoring the sorting turntable to determine if the sorting turntable is full of luggage, and determines that a baggage blockage fault has occurred, reminding the handling workers to sort the luggage as soon as possible.

10. The machine vision-based departure baggage check-in control system according to claim 7, characterized in that, Steps 4-5 further include: when the conveyor's operating output signal and operating feedback signal are inconsistent, an operating fault occurs; when the entire system is in operating mode, the vision processing module analyzes the images captured by the monitoring belt conveyor camera to determine whether the belt conveyor is running; and compares the judgment result with the signal fed back by the control module. If they are inconsistent, the vision processing module issues a fault alarm and prompts manual handling.

11. The machine vision-based departure baggage check-in control system according to claim 7, characterized in that, Steps 4-6 further include: when the downstream conveyor stops running, the system determines, through images captured by the camera, that luggage has passed by the upstream conveyor, and controls the upstream conveyor to stop and wait. After the downstream conveyor starts running, the upstream conveyor continues to run. When the entire system is in running mode, the control module receives a linkage signal for the belt conveyor to stop, and the vision processing module analyzes the images captured by the camera of the monitoring belt conveyor to determine whether luggage has passed by the belt conveyor. When luggage passes by, the vision processing module sends a signal to the control module to control the upstream belt conveyor to stop running.

Images