Machine vision-based departure luggage check-in control system

Abstract

The invention discloses a departure baggage check-in control system based on machine vision, which can optimize and simplify a baggage check-in process, improve check-in efficiency, optimize airport operation and improve passenger experience. According to the technical scheme, the system comprises a user interface module used for providing an airport worker operation and monitoring interface; the visual acquisition module is used for monitoring the state of the luggage in the check-in process in real time and capturing images and videos of the luggage in the transportation process; the visual processing module is used for collecting and processing data from the visual acquisition module and the control module, identifying the state of the luggage through image processing and algorithm analysis, and tracking possible abnormal conditions of the luggage in the transportation process in real time; and the control module is used for controlling the corresponding belt conveyor to run and stop according to the result output by the visual processing module.

Description

Technical Field

[0001] The present invention relates to the technical field of airport terminal baggage consignment, and in particular to a departure baggage consignment control system based on machine vision. Background Art

[0002] Figure 1 shows a common airport departure baggage handling system layout, such as Figure 1 As shown, the outbound baggage system at existing small and medium-sized airports typically consists of a check-in conveyor subsystem, belt conveyors, curve conveyors, and flat baggage carousels (or inclined baggage carousels). Baggage is first weighed and measured on the electronic scale conveyor (DD7) in the check-in conveyor subsystem before being sent to the X-ray security inspection conveyor (ZD7) for security screening. After passing security screening, the baggage is sent to the injection conveyor (GD7) to queue for delivery to the empty window of the collection conveyor (PD5). After collection, the baggage passes through a series of belt conveyors and a curve infeed (ZW2D) before being sent to the sorting carousel (ZP5) for sorting and delivery to the appropriate flight.

[0003] The control process of the above-mentioned existing outbound baggage system is specifically as follows.

[0004] When checking in at the check-in counter in the departure hall, departing passengers place their baggage on the weighing conveyor. The check-in staff will weigh, check the dimensions, and label the baggage before assigning it a flight number. If the baggage exceeds the permitted dimensions or weight, the passenger will hand it over to airport security for oversized baggage handling.

[0005] After labeling, standard baggage is scanned by a barcode scanner and transported from the electronic weighing conveyor to the dual-lane X-ray security inspection conveyor. The two security inspection conveyors are arranged in parallel and share one X-ray security conveyor. However, the X-ray security conveyor can only scan baggage on one of the two security inspection conveyors. Therefore, baggage cannot be on both X-ray security conveyors at the same time. In automatic operation mode, the system will ensure that only one of the two parallel X-ray security inspection conveyors has baggage on it.

[0006] After passing through the security conveyor, the baggage arrives at the end of the loading conveyor and waits. Only when the security clearance signal is received can the baggage be sent to the merge conveyor. If the signal is unsafe, the loading conveyor is locked, waiting for manual identification or baggage inspection by security personnel. The check-in conveyor line can only be restarted after the lock is released.

[0007] The merge conveyor uses window control technology with equal probability injection to prevent baggage accumulation. Each window can only allocate one bag, ensuring that each check-in counter has an equal chance of sending bags to the merge conveyor.

[0008] The baggage arriving at the merging conveyor is automatically transported to the sorting carousel for sorting baggage through a series of conveying equipment such as belt conveyors and curve conveyors. The baggage is then manually loaded into the corresponding baggage trailers or baggage containers according to the flight numbers.

[0009] The above-mentioned existing outbound baggage system has the following disadvantages.

[0010] 1. Existing equipment uses photoelectric sensors to determine whether a bag is stuck or blocked. However, if there is no photoelectric sensor at the location of the stuck bag, the system cannot make a judgment. Setting up a baggage handling system with centralized bag opening will cause the system to misjudge.

[0011] 2. When a piece of luggage is stuck in the security check machine, the system cannot make a judgment, which may cause the suspicious luggage to be missed, resulting in system misjudgment, and even in some cases, the passenger's luggage may be missed in the security check machine.

[0012] 3. Lost luggage. Accurate baggage tracking, especially the recovery of lost luggage, reduces the cost of handling irregular luggage. This prevents luggage from being sorted and not loaded onto the aircraft or being sent to the wrong flight.

[0013] 4. After the luggage arrives at the departure carousel after being checked in at the check-in counter, the handling staff needs to check it one by one to avoid omissions, which is a very labor-intensive task.

[0014] 5. Using RFID tags for luggage tracking is costly and uses non-renewable materials, which is not environmentally friendly and does not meet the requirements of sustainable development.

[0015] 6. During the use of the system, it is difficult to detect belt deviation, and when it is discovered, the belt is already severely worn. Summary of the Invention

[0016] The following is a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceivable aspects and is neither intended to identify key or critical elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that will be provided later.

[0017] The purpose of the present invention is to solve the above problems and provide a departure baggage check-in control system based on machine vision, which can optimize and simplify the baggage check-in process, improve check-in efficiency, optimize airport operations, and improve passenger experience.

[0018] The technical solution of the present invention is as follows: The present invention discloses a departure baggage check-in control system based on machine vision, the system comprising: User interface module, used to provide airport staff with an operation and monitoring interface; The visual acquisition module is used to monitor the status of baggage during check-in in real time and capture images and videos of the baggage during transportation; The visual processing module is used to collect and process data from the visual acquisition module and the control module. Through image processing and algorithm analysis, it can identify the status of the baggage and track any abnormalities that may occur during the baggage transportation process in real time. The control module is used to control the operation and stop of the corresponding belt conveyor according to the output results of the visual processing module.

[0019] According to one embodiment of the machine vision-based outbound baggage check-in control system of the present invention, the visual processing module processes the information collected by the visual acquisition module to obtain information including the luggage location and the swing arm position; and the visual processing module receives feedback signals from the control module and sends instructions to the control module based on the results of the visual processing.

[0020] According to an embodiment of the outbound baggage check-in control system based on machine vision of the present invention, the system further includes a storage module for storing historical data of the vision processing module, and the historical data is used for learning and optimizing system performance.

[0021] According to an embodiment of the outbound baggage check-in control system based on machine vision of the present invention, the specific control method of the control module includes: Bag jam: The visual processing module determines that the luggage compartment is stationary and sends a feedback signal to the control module. The control module also receives a signal that the belt conveyor is running and issues a bag jam alarm signal. Overlength: The visual processing module first identifies whether there is luggage on the electronic scale conveyor. If so, it identifies the luggage's outline and calculates its dimensions. If it meets the requirements, the next step proceeds normally. If not, an oversized luggage alarm is triggered. Standby state: The visual processing module determines whether there is luggage on the belt conveyor. If no luggage passes within the set time limit, the control module controls the belt conveyor to stop running and enter the standby state; Carousel full load: The visual processing module determines that the distance between luggage boxes on the luggage carousel is less than the set length, and sends a feedback signal to the control module, indicating that the carousel is full; Baggage left behind at the security check machine: The visual processing module recognizes that the baggage has entered the security check machine, but if it does not exit the machine within the set time limit, the control module determines that the baggage has been left behind at the security check machine and issues an alarm. Belt deviation: When the visual processing module determines that the belt is at an inconsistent distance from the guard plates on both sides and is closer to one side, it reminds the staff that maintenance is needed. When the visual processing module determines that the belt edge is severely worn, it reminds the staff that maintenance is needed.

[0022] According to an embodiment of the outbound baggage check-in control system based on machine vision of the present invention, the system is further configured to run the following check-in process: Step 1-1: After the baggage is placed on the electronic scale, the outbound baggage control system processes the images captured by the camera monitoring the electronic scale conveyor to determine whether the baggage is overlength. If the baggage is not overlength, the electronic scale weight is not overweight, and the camera detects that there is baggage on the electronic scale, the system sends a start signal to the electronic scale conveyor. The baggage stops when it reaches the level of the check-in staff. If the baggage is overlength, overweight, or not for the flight being checked, the check-in staff and the passenger will be notified at the counter to remove the baggage. Step 1-2: The baggage is tagged when the check-in staff checks in the baggage and the check-in information is sent to the outbound baggage control system; Step 1-3: The outbound baggage control system receives check-in information and determines that the baggage has been tagged. It then sends start signals to the electronic scale conveyor, security inspection machine conveyor, and injection conveyor in sequence. Based on the images captured by the cameras monitoring the electronic scale conveyor and the injection conveyor, the system determines when the baggage has left the electronic scale and stops the electronic scale conveyor. Based on the images captured by the cameras monitoring the injection conveyor, the system determines when the baggage has reached the rear end of the injection conveyor. The security inspection machine conveyor and the injection conveyor then stop, waiting for the security inspection machine's image interpretation results. Steps 1-4: The security inspection machine gives the image recognition result. The outbound baggage handling control system receives the baggage on the take-away conveyor according to the window control technology based on the image collected by the camera monitoring the belt conveyor. The baggage on the injection conveyor is first reserved at the window. When its reserved window reaches the position of the injection conveyor, the injection conveyor automatically puts the baggage into the take-away conveyor. The injection conveyor is not allowed to put baggage into the take-away conveyor at will.

[0023] According to an embodiment of the outbound baggage check-in control system based on machine vision of the present invention, the system is further configured to run the baggage tracking process as follows: Step 2-1: Camera calibration: Calibrate each camera to obtain its intrinsic and extrinsic parameters; Step 2-2: Image preprocessing and feature extraction: Image preprocessing refers to the preprocessing of the images captured by each camera, including denoising, grayscale conversion, and binarization. Feature extraction refers to the use of feature extraction algorithms to extract points or areas with obvious features from the image. Step 2-3: Image matching and target tracking: Image matching refers to finding similar feature points between images captured by different cameras using a feature point matching algorithm. Target tracking refers to continuously tracking the target using a target tracking algorithm based on the matched feature points. Step 2-4: Multi-camera data fusion and tracking optimization: Data fusion refers to fusing the tracking results from different cameras, and tracking optimization refers to further optimizing the tracking results using optimization algorithms; Steps 2-5: The outbound baggage control system converts the object position from the camera's perspective into the global coordinate system, continuously tracks the object in each camera, and generates the object's motion trajectory. The object trajectories from different cameras are then associated to form a cross-camera object motion trajectory. By matching the baggage's feature information and utilizing the position information in the global coordinate system, the baggage trajectory association and tracking are achieved.

[0024] According to an embodiment of the outbound baggage check-in control system based on machine vision of the present invention, the system is further configured to run the baggage opening room control process as follows: Step 3-1: The outbound baggage control system determines whether there is any baggage on the belt conveyor based on the images collected by the camera monitoring the baggage unpacking room; Step 3-2: When a bag is on the belt conveyor, the system determines whether it is qualified or suspicious based on the baggage's image recognition information recorded by the system. Step 3-3: The check-in channel security machine identifies the baggage as suspicious and checks whether the diverter arm is in the extended position. Step 3-4: The check-in channel security machine determines that the baggage is standard luggage and determines whether the diverter arm is in the return position.

[0025] According to an embodiment of the outbound baggage check-in control system based on machine vision of the present invention, the system is further configured to run the conveyor control process as follows: Step 4-1: Start the conveyor: After the staff presses the start button, the system uses the image captured by the camera to determine that there is no luggage on the conveyor belt, and then starts it; Step 4-2: Stopping the conveyor: After the staff presses the stop button, the system uses the image captured by the camera to determine that there is no luggage on the conveyor belt, and then stops the operation; Step 4-3: Energy-saving processing: The visual processing module analyzes the images captured by the visual acquisition module's camera, determines whether there is luggage on each section of the conveyor belt, and records the time. When the visual processing module recognizes that no luggage passes through the conveyor line within the set time, it sends an energy-saving signal to the control module, which controls the belt conveyors to stop running in sequence, and the entire system enters energy-saving standby mode. In energy-saving standby mode, the visual processing module analyzes the images captured by the visual acquisition module-injector area camera, determines that there is luggage on the injection conveyor, and sends an energy-saving standby mode termination signal to the control module, which controls the belt conveyors to start in sequence; Step 4-4: Bag jam handling: When the conveyor is running, the system uses the images captured by the camera to determine that a piece of luggage on the conveyor belt has not moved for a long time. The system determines that a bag jam has occurred, controls the conveyor to stop, and issues an alarm. After the bag jam is cleared, press the reset button on the electrical control cabinet to resume the system. When the sorting carousel is running, the system uses the images captured by the camera monitoring the sorting carousel to determine that the sorting carousel is full of luggage. This determines that a bag jam has occurred and reminds the handlers to sort the luggage as soon as possible. Step 4-5: Operational Failure and Safeguard Processing: When the conveyor's operating output signal and operating feedback signal are inconsistent, an operational failure occurs. When the entire system is in operation mode, the visual processing module analyzes the image captured by the monitoring belt conveyor camera to determine whether the belt conveyor is operating. The judgment result is compared with the signal fed back by the control module. If there is any inconsistency, the visual processing module will issue a fault alarm and prompt manual processing. Steps 4-6: Queue processing: When the downstream conveyor stops, the system uses the images captured by the camera to determine if there is luggage passing through the upstream conveyor. The system controls the upstream conveyor to stop and wait. After the downstream conveyor starts running, the upstream conveyor continues to run. When the entire system is in running mode, the control module receives the linkage signal for the belt conveyor to stop. The visual processing module analyzes and monitors the images captured by the camera of the previous belt conveyor to determine whether there is luggage passing through the belt conveyor. If there is luggage passing through, the visual processing module sends a signal to the control module to control the previous belt conveyor to stop running. Step 4-7: Reset and start of emergency stop: After the emergency stop switch is pulled up, the conveyor should not start immediately. Only after pressing the reset button of the control cabinet, the conveyor will start step by step.

[0026] Compared to existing technologies, the present invention has the following advantages: The machine vision-based outbound baggage check-in control system of the present invention comprises a user interface module, a visual acquisition module, a visual processing module, and a control module. First, the system of the present invention eliminates components such as photoelectric sensors, foot switches, barcode scanners, and limit switches found in existing systems, reducing the likelihood of safety incidents caused by component aging, wear, or failure, thereby optimizing system components and improving reliability. Second, by using the system of the present invention to control the outbound baggage check-in process, check-in staff no longer need to step on a foot pedal; they can simply attach a baggage tag and wait for the system to automatically deliver the baggage to the sorting carousel. Handlers at the sorting carousel no longer need to verify baggage information; the system automatically verifies and issues an alarm if any anomalies are detected, streamlining the workflow. Third, the system of the present invention enables full-process tracking of checked baggage, reducing workload and improving system efficiency. Fourth, the system of the present invention can promptly detect equipment changes, enabling preemptive maintenance and upkeep, thereby extending system lifespan and reducing the number of emergency repairs, thus reducing maintenance pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above features and advantages of the present invention will be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or characteristics may have the same or similar reference numerals.

[0028] Figure 1 The figure shows the layout of the existing airport departure baggage system.

[0029] Figure 2 The figure shows a principle diagram of an embodiment of a machine vision-based outbound baggage check-in control system of the present invention.

[0030] Figure 3 The diagram shows the layout of the centralized unpacking outbound baggage system.

[0031] Figure 4 Shown Figure 2 The diagram shows the check-in process of the system running.

[0032] Figure 5 Shown Figure 2 Schematic diagram of the baggage tracking process operated by the system shown.

[0033] Figure 6 Shown Figure 2 The diagram shows the control process of the package opening room of the system operation.

[0034] Figure 7 Shown Figure 2 Schematic diagram of the conveyor control process of the system operation is shown. DETAILED DESCRIPTION

[0035] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Note that the various aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention.

[0036] Figure 2 The principle of an embodiment of the outbound baggage check-in control system based on machine vision of the present invention is shown. Figure 2 The system of this embodiment includes: a user interface module, a visual acquisition module, a visual processing module, and a control module. In addition, the system also includes a storage module. Figure 3 The layout of the centralized unpacking outbound baggage system is shown in the figure. Figure 3In the diagram, SX represents a camera, PD represents a belt conveyor, ZW represents a curve conveyor, ZP represents a sorting carousel, FLQ represents a swing arm diverter (vertical diverters and swing wheel diverters are also applicable), DD7 represents an electronic scale conveyor, ZD7 represents an inspection machine conveyor, GD7 represents an injection conveyor, and D-1 and D-2 represent universal ball platforms. The cameras are categorized as follows: SX1, SX2, SX3, and SX4 for monitoring electronic scale conveyors; SX5, SX10, SX11, and SX15 for monitoring belt conveyors; SX6, SX7, SX8, and SX9 for monitoring injection conveyors; SX12, SX13, and SX14 for monitoring the bag opening room; and SX16, SX17, SX18, and SX19 for monitoring the sorting carousel. The layout of each airport is different, and so are the belt conveyor arrangements. Figure 3 For matching only Figure 2 The examples of system composition are only for the convenience of explanation.

[0037] The user interface module is used to provide airport staff with an operation and monitoring interface to display baggage status, abnormal alarm information, etc.

[0038] The visual acquisition module is used to monitor the status of the baggage check-in process in real time and capture images and videos of the baggage during transportation. The visual acquisition module includes Figure 3 The data acquisition performed by the system of this embodiment does not require sensors in traditional systems, such as photoelectric sensors, limit sensors, etc.

[0039] The Vision Processing Module collects and processes data from the Vision Acquisition Module and the Control Module. Through image processing and algorithm analysis, it identifies the baggage's status and tracks any anomalies that may arise during transportation in real time. The Vision Processing Module processes the information collected by the Vision Acquisition Module to determine information such as baggage location and arm position. Furthermore, the Vision Processing Module receives feedback from the Control Module, integrates the visual processing results, and issues instructions to the Control Module.

[0040] The storage module also stores historical data from the visual processing module for learning and optimizing system performance. The visual processing module's hardware and software components are the visual processing software and host computer system installed on the server, while the storage module's hardware components are the server's storage hard drive.

[0041] Examples of algorithms used in the visual processing module to identify baggage status include the Faster-RCNN algorithm, the YOLO series of algorithms, and the Mask-RCNN algorithm. Examples of algorithms used in the visual processing module to track baggage status in real time include deep learning-based target tracking algorithms, correlation filtering-based target tracking algorithms, optical flow-based target tracking algorithms, and Siamese network (Twin Network) tracking algorithms.

[0042] The control module is used to control the operation and stop of the corresponding belt conveyor according to the output of the visual processing module. The hardware equipment of the control module is the PLC controller and related input and output modules, which are located in the electrical control box.

[0043] The specific control method of the control module is as follows.

[0044] Bag jam situation: The visual processing module determines that the suitcase is stationary and gives a feedback signal to the control module. At the same time, the control module receives a signal that the belt conveyor is running, and the control module gives a bag jam alarm signal.

[0045] Oversized baggage: The visual processing module first identifies whether there is any luggage on the electronic scale conveyor. If so, it identifies the luggage's outline and calculates its dimensions. If it meets the regulations, the normal next step will be taken. If it does not, an oversized baggage alarm will be issued.

[0046] Standby state: The visual processing module determines whether there is luggage on the belt conveyor. If no luggage passes within a set time limit (for example, 10 minutes, the time is adjustable), the control module controls the belt conveyor to stop running and enter the standby state.

[0047] Carousel full load: The visual processing module determines that the distance between luggage boxes on the luggage carousel is less than the set length (for example, 1 meter), and sends a feedback signal to the control module, indicating that the carousel is full.

[0048] Baggage left behind in the security check machine: The visual processing module recognizes that the baggage has entered the security check machine, but the baggage has not come out of the security check machine after a set time limit (for example, 10 seconds). The control module determines that the baggage has been left behind in the security check machine and issues an alarm.

[0049] Belt deviation: If the visual processing module determines that the belt is not aligned with the guard plates on both sides and is closer to one side (exceeding the initial limit), it alerts the operator that maintenance is required. If the visual processing module determines that the belt edge is severely worn, it alerts the operator that maintenance is required.

[0050] The outbound baggage check-in control system based on machine vision of this embodiment is configured to allow the following workflows: check-in process, baggage tracking process, baggage opening room control process, and conveyor control process.

[0051] Combine Figure 3 As shown, the above processes are described in more detail below.

[0052] Figure 4 The check-in process of the system operation is shown and is described in detail below.

[0053] Step 1-1: After the baggage is placed on electronic scale DD7, the outbound baggage control system processes the image captured by camera SX1 on the electronic scale conveyor to determine whether the baggage is overlength. If the baggage is not overlength, the weight on the electronic scale is not overweight, and camera SX1 detects a baggage on the electronic scale, the system signals the electronic scale conveyor to start. The system stops when the baggage reaches the level of the check-in staff. For baggage that is overlength, overweight, or not for a checked flight, the check-in staff and the passenger are prompted to remove the baggage at the counter.

[0054] Step 1-2: When the baggage is checked in by the check-in staff, a baggage tag is attached to the baggage and the check-in information (flight information, passenger information, baggage information, etc.) is sent to the outbound baggage control system.

[0055] Step 1-3: The outbound baggage control system receives check-in information and determines that the baggage has been tagged. It then sequentially sends start signals to the electronic scale conveyor, security inspection conveyor, and infeed conveyor. Based on images captured by cameras SX1 and SX6, which monitor the infeed conveyor, the electronic scale conveyor stops when the baggage leaves the electronic scale. Based on images captured by camera SX6, which monitors the infeed conveyor, the security inspection conveyor and infeed conveyor stop when the baggage reaches the rear end of the infeed conveyor. Wait for the security inspection machine's image interpretation results.

[0056] Steps 1-4: The security inspection machine generates image recognition results. The outbound baggage handling control system, based on images captured by cameras SX5 and SX11 monitoring the belt conveyors, accepts baggage on take-away conveyor PD5-0 using window control technology. Baggage on the take-away conveyor is first reserved for a window. When the reserved window reaches the take-away conveyor, the take-away conveyor automatically deposits the baggage onto the take-away conveyor. The take-away conveyor is not allowed to randomly deposit baggage onto the take-away conveyor. Window control technology ensures that baggage enters the take-away conveyor at an equal rate and pitch.

[0057] Figure 5 The baggage tracking process of the system is shown and described in detail below.

[0058] Step 2-1: Camera calibration. Each camera is calibrated to obtain its intrinsic parameters (such as focal length and optical center) and extrinsic parameters (such as the camera's position and orientation in 3D space). The calibration process typically involves photographing a calibration plate of known dimensions and extracting feature points on the plate to calculate the camera parameters.

[0059] After the parameters are calculated, it is best to optimize them: through iterative optimization algorithms, the accuracy of camera parameters can be further improved, and the consistency of parameters between cameras can be ensured to reduce errors in subsequent image matching and fusion processes.

[0060] Step 2-2: Image preprocessing and feature extraction.

[0061] Image preprocessing refers to the preprocessing of images captured by each camera, including steps such as denoising, grayscale conversion, and binarization. The preprocessing process aims to improve image quality and reduce the computational complexity and errors of subsequent processing.

[0062] Feature extraction refers to the use of feature extraction algorithms (such as SIFT, SURF, ORB, etc.) to extract points or regions with obvious features from an image. These feature points or regions will play a key role in the subsequent image matching process.

[0063] Step 2-3: Image matching and target tracking.

[0064] Image matching involves finding similar feature points between images captured by different cameras using a feature point matching algorithm (such as FLANN and BFMatcher). The matching process takes into account factors such as the position, orientation, and scale of the feature points to ensure accurate matching.

[0065] Target tracking involves continuously tracking a target based on matched feature points using a target tracking algorithm (such as Kalman filtering, particle filtering, or DeepSORT). Tracking algorithms must continuously update information such as the target's position and velocity to accommodate dynamic changes in the target.

[0066] Steps 2-4: Multi-camera data fusion and tracking optimization.

[0067] Data fusion combines tracking results from different cameras to obtain more accurate and complete tracking information. The fusion process needs to take into account factors such as the positional relationship between cameras and overlapping areas to ensure data accuracy and consistency.

[0068] Tracking optimization refers to further optimizing the tracking results using optimization algorithms (such as genetic algorithms, particle swarm optimization, etc.). The optimization process aims to reduce tracking errors and improve tracking stability and accuracy.

[0069] Steps 2-5: The outbound baggage control system converts the object's position from the camera's perspective into the global coordinate system. It continuously tracks the object in each camera and generates its motion trajectory. The object's trajectories from different cameras are then linked to form a cross-camera trajectory. By matching baggage feature information (such as color, shape, texture, flight information, passenger information, and security inspection image information), the global coordinate system is used to link and track the baggage's trajectory.

[0070] Figure 6 The control process of the package opening room of the system operation is shown and is described in detail as follows.

[0071] Step 3-1: The outbound baggage control system determines whether there is any baggage arriving on the belt conveyor PD5-1 based on the images collected by the cameras SX12 and SX14 monitoring the baggage unpacking room.

[0072] a) The visual processing module analyzes the images captured by the visual acquisition module (i.e., the camera monitoring the bag opening room) in real time to determine whether there is any luggage on the belt conveyor PD5-1. If there is no luggage, no action is taken.

[0073] b) When the visual processing module determines that there is luggage, it determines the luggage status based on the luggage image information fed back by the control module;

[0074] c) The visual processing module determines that the baggage is normal and determines the position of the diverter FLQ's swing arm. If the swing arm is in the retracted position, the visual processing module and the control module do not take any action. If the swing arm is in the extended position, the visual processing module sends a command to the control module, which first stops conveyor belt PD5-1, then controls the swing arm to swing back, and then restarts conveyor belt PD5-1.

[0075] d) The visual processing module identifies the bag as suspicious and determines the position of the diverter FLQ's swing arm. When the swing arm is in the extended position, the visual processing module and the control module remain inactive. When the swing arm is in the retracted position, the visual processing module issues a command to the control module, which first stops conveyor belt PD5-1, then controls the swing arm to swing outward. After the swing arm is outward, conveyor belt PD5-1 is restarted.

[0076] e) The suspicious baggage passes through belt conveyor PD5-1 and belt conveyor PD5-2 in sequence and arrives at universal ball platform D-1 for on-site inspection.

[0077] Step 3-2: When there is a bag on the belt conveyor PD5-1, the bag is judged as qualified baggage or suspicious baggage based on the image recognition information of the bag recorded by the system.

[0078] Step 3-3: The check-in channel security machine identifies the baggage as suspicious and determines whether the swing arm of the diverter FLQ is in the extended position.

[0079] a) The swing arm of diverter FLQ is in the extended position, and belt conveyor PD5-1 continues to run. The baggage passes through diverter FLQ and conveyor belt PD5-2 and reaches universal ball platform D-1.

[0080] b) The diverter FLQ's swing arm is in the retracted position. The system determines whether there is any luggage within the diverter's swing arm range on belt conveyor PD5-4. If there is no luggage, the system controls the diverter FLQ's swing arm to swing outward, and the luggage passes through diverter FLQ and conveyor belt PD5-2 to universal ball transfer platform D-1. If there is luggage, belt conveyor PD5-1 stops and waits until there is no luggage on belt conveyor PD5-4. The system then controls the diverter FLQ's swing arm to swing outward, and belt conveyor PD5-1 starts to operate, passing through diverter FLQ and conveyor belt PD5-2 to universal ball transfer platform D-1.

[0081] c) The luggage on the universal ball platform D-1 is waiting for on-site inspection.

[0082] d) The re-inspection personnel place the baggage that has passed the manual inspection onto security inspection conveyor ZD7. The outbound baggage handling control system uses the images captured by camera SX13, which monitors the baggage unpacking area, to determine whether there is baggage on security inspection conveyor ZD7. If so, the system controls security inspection conveyor ZD7 and belt conveyor PD5-3 to operate. Based on the images captured by camera SX14, which monitors the baggage unpacking area, the system controls security inspection conveyor ZD7 and belt conveyor PD5-3 to stop when the baggage has reached the rear end of belt conveyor PD5-3. The system then waits for the image recognition results from the security inspection system in the baggage unpacking area.

[0083] e) The image recognition result from the baggage opening room security inspection machine is still suspicious, prompting on-site re-inspection personnel to remove the baggage for re-inspection. The image recognition result is normal. Based on the images captured by camera SX14 monitoring the baggage opening room and camera SX15 monitoring the belt conveyor, the system determines that there is no bag collision, controls belt conveyor PD5-3 to operate, and puts the baggage onto belt conveyor PD5-4.

[0084] Step 3-4: The check-in channel security machine determines that the baggage is standard and determines whether the swing arm of the diverter FLQ is in the return position.

[0085] a) The swing arm of the diverter FLQ is in the swing-back position, the belt conveyor PD5-1 continues to run, and the baggage is sent to the sorting carousel via the conveyor belts PD5-4 and PD5-5, waiting for manual sorting.

[0086] b) The swing arm of diverter FLQ is in the extended position. The outbound baggage handling control system uses images captured by camera SX14 monitoring the baggage unpacking area and camera SX15 monitoring the belt conveyor to determine whether there is any baggage within the diverter's swing arm range on belt conveyor PD5-4. If there is no baggage, the system controls the swing arm of diverter FLQ to swing back. If there is baggage, belt conveyor PD5-1 stops and waits until there is no baggage within the diverter's swing arm range on belt conveyor PD5-4. The system then controls the swing arm of diverter FLQ to swing back, allowing belt conveyor PD5-1 to operate.

[0087] Figure 7 The conveyor control process of the system operation is shown and is described in detail below.

[0088] Step 4-1: Start the conveyor.

[0089] After the operator presses the start button, the system detects the absence of luggage on the conveyor belts based on the camera image and activates the system. The activation sequence is from downstream to upstream. The downstream conveyor starts first, and after that, the upstream conveyor starts with a configurable delay.

[0090] A staff member presses the system start button in the control module, which sends a command to the visual processing module. The visual processing module analyzes the images captured by the visual acquisition module's camera to determine whether there is luggage on each section of the conveyor belt. If there is no luggage on the entire conveyor line, the visual processing system sends a start signal to the control module, which starts the conveyor lines one second apart, starting from the bottom up. If there is luggage on the entire conveyor line, the visual processing system issues an alarm, prompting manual intervention.

[0091] Step 4-2: Stopping the conveyor.

[0092] When the operator presses the stop button, the system uses the camera's imagery to determine that there are no luggage on the conveyor belts and stops operation. The system's shutdown sequence is from upstream to downstream. The upstream conveyor stops first, and after that, the downstream conveyor stops with a configurable delay to ensure the luggage is cleared.

[0093] The operator presses the system stop button in the control module, which sends a command to the visual processing module. The visual processing module analyzes the images captured by the visual acquisition module's camera to determine whether there is luggage on each conveyor belt section. If the visual processing module determines that there is no luggage on this belt conveyor and all upstream belt conveyors, the visual processing system sends a stop signal to the control module, which stops the conveyor belt section after 1 second.

[0094] Step 4-3: Energy saving processing.

[0095] The system uses images captured by the camera to determine if there are no bags on the conveyor belt for a certain period of time (adjustable as needed). The waiting time for the conveyor to enter energy-saving mode is a user-adjustable parameter, allowing users to set it based on their usage. The control module also implements effective measures to prevent false signal interference and the impact of energy-saving functions. When the system needs to enter energy-saving mode, the conveyor will delay stopping from top to bottom to enter energy-saving mode. When the system is awakened, the conveyor will delay starting from bottom to top.

[0096] The visual processing module analyzes images captured by the visual acquisition module's camera to determine whether there is luggage on each section of the conveyor belt and record the time. If the visual processing module detects that no luggage has passed through the conveyor line for 20 minutes, it sends an energy-saving signal to the control module. The control module then stops the conveyor belts one second at a time, sequentially from top to bottom, and the entire system enters energy-saving standby mode.

[0097] In energy-saving standby mode, the visual processing module analyzes the images collected by the visual acquisition module-the injection machine area camera, determines that there is luggage on the injection conveyor, and sends an energy-saving standby mode termination signal to the control module; the control module controls the belt conveyor to start in sequence from top to bottom at intervals of 1 second.

[0098] Step 4-4: Packet blocking processing.

[0099] When the conveyor is running, the system uses the images captured by the camera to determine if a piece of luggage on the conveyor belt has been immobile for an extended period. This indicates a baggage jam, causing the conveyor to shut down and an alarm to sound. Once the jam is cleared, press the reset button on the electrical control cabinet to resume system operation.

[0100] When the sorting carousel is running, the system monitors the images captured by the sorting carousel cameras SX16, SX17, SX18, and SX19. If the sorting carousel is full of luggage, it will be judged as a bag jam and the baggage handling staff will be reminded to sort the luggage as soon as possible.

[0101] Step 4-5: Operational failure and maintenance processing.

[0102] When the conveyor operation output signal and the operation feedback signal are inconsistent, an operation fault occurs.

[0103] When the entire system is in operation mode, the visual processing module analyzes the images collected by the monitoring camera of the belt conveyor to determine whether the belt conveyor is running; and compares the judgment result with the signal feedback from the control module. If there is any inconsistency, the visual processing module will issue a fault alarm and prompt manual processing.

[0104] Steps 4-6: Queue processing.

[0105] When the downstream conveyor stops running (due to baggage jam, malfunction, maintenance, queue, etc.), the system uses the images collected by the camera to determine that there is baggage passing through the upstream conveyor. The system controls the upstream conveyor to stop and wait. After the downstream conveyor starts running, the upstream conveyor continues to run.

[0106] When the entire system is in operation mode, the control module receives the linkage signal for the belt conveyor to stop, and the visual processing module analyzes and monitors the image captured by the camera of the previous section of the belt conveyor to determine whether there is luggage passing by the belt conveyor; if there is luggage passing by, the visual processing module sends a signal to the control module to control the previous section of the belt conveyor to stop running.

[0107] Steps 4-7: Reset and start the emergency stop.

[0108] After the emergency stop switch is pulled up, the conveyor should not start immediately. Only after pressing the reset button of the control cabinet, the conveyor will start step by step.

[0109] In addition, the system of this embodiment also realizes the following functions.

[0110] 1. When the system determines through collected images that the conveyor belt is deviating, it reminds airport maintenance personnel to promptly maintain the corresponding belt conveyor.

[0111] 2. When the system determines through collected images that luggage is lost or left behind before reaching the sorting carousel, it will remind airport staff to check it in time.

[0112] 3. The system automatically compares baggage information handled at the check-in counter with that leaving the sorting carousel. If any discrepancies are detected, on-site staff will be notified to avoid disrupting passengers' itineraries.

[0113] 4. The system will retain photos of the luggage passing through each section of the conveyor belt, so that passengers can check them if they encounter any problems later.

[0114] In general, the system of the present invention has the following technical effects.

[0115] 1. Optimize and simplify the baggage check-in process, improve check-in efficiency, and optimize airport operations:

[0116] Improve resource utilization: Improved check-in efficiency means that more passengers can be processed per unit time, making better use of resources such as check-in counters and self-service check-in equipment, improving the efficiency of airport facilities and reducing idle resources and waste.

[0117] Accelerate passenger flow: The fast check-in process enables passengers to flow faster from the check-in area to the waiting area and boarding gate, speeding up the flow of passengers within the airport, helping to alleviate airport congestion, improve the overall operational efficiency of the airport, and ensure the smooth operation of the airport.

[0118] Improve collaborative operation capabilities: An efficient check-in process can better connect and work together with other airport processes (such as security checks, waiting areas, etc.), forming an efficient and smooth operation system, improving the overall operational efficiency and service quality of the airport, and reducing problems such as flight delays caused by poor connections between various links.

[0119] Realize the whole process tracking of passenger baggage check-in: check baggage like checking express delivery - scan the code / enter the baggage number to display the node and estimated carousel arrival time in real time, reducing the error rate of passenger baggage.

[0120] 2. Improve passenger experience:

[0121] Reduced waiting time: With the improvement of check-in efficiency, the time passengers wait in line for check-in procedures is greatly shortened. Passengers can quickly complete check-in and enter the waiting area, making the travel process smoother and easier, reducing travel fatigue and anxiety.

[0122] Improve travel satisfaction: Fast and convenient check-in processes and real-time display of baggage location will give passengers a good impression of airport services, increase their satisfaction with the entire travel process, help build a good brand image, and make passengers more willing to choose this airport for travel

[0123] Increased travel flexibility: Improved check-in efficiency allows passengers more time to shop, dine, or rest at the airport, enriching their airport experience. It also provides passengers with more time to deal with possible emergencies (such as the need to reschedule their itinerary after a flight delay), thereby increasing travel flexibility.

[0124] Although the above methods are illustrated and described as a series of acts for simplicity of explanation, it is to be understood and appreciated that these methods are not limited by the order of the acts, as some acts may occur in a different order and / or concurrently with other acts from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art according to one or more embodiments.

[0125] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A machine vision-based outbound baggage check-in control system, characterized in that: The system includes: User interface module, used to provide airport staff with an operation and monitoring interface; The visual acquisition module is used to monitor the status of baggage during check-in in real time and capture images and videos of the baggage during transportation; The visual processing module is used to collect and process data from the visual acquisition module and the control module. Through image processing and algorithm analysis, it can identify the status of the baggage and track any abnormalities that may occur during the baggage transportation process in real time. The control module is used to control the operation and stop of the corresponding belt conveyor according to the output results of the visual processing module.

2. The outbound baggage check-in control system based on machine vision according to claim 1, characterized in that: On the one hand, the visual processing module processes the information collected by the visual acquisition module to obtain information including the luggage location and swing arm position; on the other hand, the visual processing module receives the signal feedback from the control module and sends instructions to the control module based on the results of the integrated visual processing.

3. The outbound baggage check-in control system based on machine vision according to claim 1, characterized in that: The system also includes a storage module for storing historical data of the visual processing module, and the historical data is used for learning and optimizing system performance.

4. The outbound baggage check-in control system based on machine vision according to claim 1, characterized in that: The specific control methods of the control module include: Bag jam: The visual processing module determines that the luggage compartment is stationary and sends a feedback signal to the control module. The control module also receives a signal that the belt conveyor is running and issues a bag jam alarm signal. Overlength: The visual processing module first identifies whether there is luggage on the electronic scale conveyor. If so, it identifies the luggage's outline and calculates its dimensions. If it meets the requirements, the next step proceeds normally. If not, an oversized luggage alarm is triggered. Standby state: The visual processing module determines whether there is luggage on the belt conveyor. If no luggage passes within the set time limit, the control module controls the belt conveyor to stop running and enter the standby state; Carousel full load: The visual processing module determines that the distance between luggage boxes on the luggage carousel is less than the set length, and sends a feedback signal to the control module, indicating that the carousel is full; Baggage left behind at the security check machine: The visual processing module recognizes that the baggage has entered the security check machine, but the baggage has not been removed from the security check machine within the set time limit. The control module determines that the baggage has been left behind at the security check machine and issues an alarm. Belt deviation: When the visual processing module determines that the belt is at an inconsistent distance from the guard plates on both sides and is closer to one side, it reminds the staff that maintenance is needed. When the visual processing module determines that the belt edge is severely worn, it reminds the staff that maintenance is needed.

5. The outbound baggage check-in control system based on machine vision according to claim 1, characterized in that: The system is also configured to run the check-in process as follows: Step 1-1: After the baggage is placed on the electronic scale, the outbound baggage control system processes the images captured by the camera monitoring the electronic scale conveyor to determine whether the baggage is overlength. If the baggage is not overlength, the electronic scale weight is not overweight, and the camera detects that there is baggage on the electronic scale, the system sends a start signal to the electronic scale conveyor. The baggage stops when it reaches the level of the check-in staff. If the baggage is overlength, overweight, or not for the flight being checked, the check-in staff and the passenger will be notified at the counter to remove the baggage. Step 1-2: The baggage is tagged when the check-in staff checks in the baggage and the check-in information is sent to the outbound baggage control system; Step 1-3: The outbound baggage control system receives check-in information and determines that the baggage has been tagged. It then sends start signals to the electronic scale conveyor, security inspection machine conveyor, and injection conveyor in sequence. Based on the images captured by the cameras monitoring the electronic scale conveyor and the injection conveyor, the system determines when the baggage has left the electronic scale and stops the electronic scale conveyor. Based on the images captured by the cameras monitoring the injection conveyor, the system determines when the baggage has reached the rear end of the injection conveyor. The security inspection machine conveyor and the injection conveyor then stop, waiting for the security inspection machine's image interpretation results. Steps 1-4: The security inspection machine gives the image recognition result. The outbound baggage handling control system receives the baggage on the take-away conveyor according to the window control technology based on the image collected by the camera monitoring the belt conveyor. The baggage on the injection conveyor is first reserved at the window. When its reserved window reaches the position of the injection conveyor, the injection conveyor automatically puts the baggage into the take-away conveyor. The injection conveyor is not allowed to put baggage into the take-away conveyor at will.

6. The outbound baggage check-in control system based on machine vision according to claim 1, characterized in that: The system is also configured to run the baggage tracking process as follows: Step 2-1: Camera calibration: Calibrate each camera to obtain its intrinsic and extrinsic parameters; Step 2-2: Image preprocessing and feature extraction: Image preprocessing refers to the preprocessing of the images captured by each camera, including denoising, grayscale conversion, and binarization. Feature extraction refers to the use of feature extraction algorithms to extract points or areas with obvious features from the image. Step 2-3: Image matching and target tracking: Image matching refers to finding similar feature points between images captured by different cameras using a feature point matching algorithm. Target tracking refers to continuously tracking the target using a target tracking algorithm based on the matched feature points. Step 2-4: Multi-camera data fusion and tracking optimization: Data fusion refers to fusing the tracking results from different cameras, and tracking optimization refers to further optimizing the tracking results using optimization algorithms; Steps 2-5: The outbound baggage control system converts the object position from the camera's perspective into the global coordinate system, continuously tracks the object in each camera, and generates the object's motion trajectory. The object trajectories from different cameras are then associated to form a cross-camera object motion trajectory. By matching the baggage's feature information and utilizing the position information in the global coordinate system, the baggage trajectory association and tracking are achieved.

7. The outbound baggage check-in control system based on machine vision according to claim 1, characterized in that: The system is also configured to run the package opening room control process as follows: Step 3-1: The outbound baggage control system determines whether there is any baggage on the belt conveyor based on the images collected by the camera monitoring the baggage unpacking room; Step 3-2: When a bag is on the belt conveyor, the system determines whether it is qualified or suspicious based on the baggage's image recognition information recorded by the system. Step 3-3: The check-in channel security machine identifies the baggage as suspicious and checks whether the diverter arm is in the extended position. Step 3-4: The check-in channel security machine determines that the baggage is standard luggage and determines whether the diverter arm is in the return position.

8. The outbound baggage check-in control system based on machine vision according to claim 1, characterized in that: The system is also configured to run the conveyor control process as follows: Step 4-1: Start the conveyor: After the staff presses the start button, the system uses the image captured by the camera to determine that there is no luggage on the conveyor belt, and then starts it; Step 4-2: Stopping the conveyor: After the staff presses the stop button, the system uses the image captured by the camera to determine that there is no luggage on the conveyor belt, and then stops the operation; Step 4-3: Energy saving processing; Step 4-4: Packet blocking processing; Step 4-5: Operational failure and safeguard handling; Step 4-6: Queue processing; Step 4-7: Reset and start of emergency stop: After the emergency stop switch is pulled up, the conveyor should not start immediately. Only after pressing the reset button of the control cabinet, the conveyor will start step by step.

9. The outbound baggage check-in control system based on machine vision according to claim 8, characterized in that: Step 4-3 further includes: the visual processing module analyzes the image captured by the camera of the visual acquisition module, determines whether there is luggage on each section of the conveyor belt and records the time; when the visual processing module recognizes that no luggage passes through the conveyor line within the set time, it sends an energy-saving signal to the control module, and the control module controls the belt conveyors to stop running in sequence, and the entire system enters the energy-saving standby mode. In the energy-saving standby mode, the visual processing module analyzes the image captured by the camera of the visual acquisition module-injection machine area, determines that there is luggage on the injection conveyor, and the visual processing module sends an energy-saving standby mode termination signal to the control module, and the control module controls the belt conveyors to start in sequence.

10. The outbound baggage check-in control system based on machine vision according to claim 8, characterized in that: Step 4-4 further includes: when the conveyor is in operation, the system determines through images captured by the camera that a piece of luggage on the conveyor belt has been stationary for a long time, the system determines that a baggage jam has occurred, controls the conveyor to stop and issues an alarm, and after the baggage jam has been cleared, presses the reset button on the electrical control cabinet to resume system operation; when the sorting carousel is in operation, the system determines through images captured by the camera monitoring the sorting carousel that the sorting carousel is full of luggage, determines that a baggage jam has occurred, and reminds the porter to sort the luggage as soon as possible.

11. The outbound baggage check-in control system based on machine vision according to claim 8, characterized in that: Steps 4-5 further include: when the conveyor operation output signal and the operation feedback signal are inconsistent, an operation fault occurs; when the entire system is in operation mode, the visual processing module analyzes the image captured by the monitoring belt conveyor camera to determine whether the belt conveyor is running; and compares the judgment result with the signal feedback from the control module. If there is inconsistency, the visual processing module will issue a fault alarm and prompt manual processing.

12. The outbound baggage check-in control system based on machine vision according to claim 8, characterized in that: Steps 4-6 further include: when the downstream conveyor stops running, the system determines through the image captured by the camera that there is luggage passing through the upstream conveyor, and the system controls the upstream conveyor to stop and wait. After the downstream conveyor starts running, the upstream conveyor continues to run; when the entire system is in running mode, the control module receives the linkage signal for the belt conveyor to stop, and the visual processing module analyzes and monitors the image captured by the camera of the previous section of the belt conveyor to determine whether there is luggage passing through the belt conveyor; if there is luggage passing through, the visual processing module sends a signal to the control module to control the previous section of the belt conveyor to stop running.

Images