An airport passenger carry-on luggage visual measurement method

Abstract

The application provides an airport passenger hand luggage visual measurement method, comprising: arranging a depth camera above and on the side of the security check; collecting the image of the passenger and the hand luggage through the depth camera, and judging the number of hand luggage according to the collected image; generating a single-view three-dimensional point cloud image according to the image, and then fusing and splicing the double-view local point cloud image to generate a three-dimensional point cloud image and perform three-dimensional reconstruction; projecting or separating the model of the hand luggage information based on the three-dimensional reconstruction model, and calculating the size of the luggage in the model; judging whether the size of the luggage or the number of the luggage exceeds the preset threshold, and if so, prompting the passenger to process. The measurement method has the advantages of high accuracy and high speed, and can measure the number and size of the hand luggage carried by the passenger without affecting the security check efficiency of the passenger, improve the management level of the three-exceeding hand luggage of the passenger, improve the flight punctuality rate, and reduce the operating cost of the airline.

Description

Technical Field

[0001] This invention relates to the field of civil aviation, and in particular to a visual measurement method for airport passenger carry-on baggage. Background Technology

[0002] This invention relates to the current situation at domestic and international airports where passengers need to check in large luggage, while small carry-on luggage is placed in the overhead bins. However, in recent years, carry-on luggage has often been found to be oversized, overweight, and oversized. After these three issues, the overhead bins cannot hold more luggage. The excess luggage not only affects security check efficiency but also poses a safety hazard to the cabin.

[0003] Existing machine vision technology can be used to measure the dimensions of mechanical parts, engines, etc., and can also be applied to the measurement of baggage dimensions, such as the invention patents CN111353985B, CN110838142B and the invention publication CN113418467A. However, these visual recognition technologies are mostly aimed at checked baggage placed on conveyor belts or fixed surfaces, while carry-on baggage has more diverse forms, more varied states, more complex backgrounds, and many difficulties such as passenger movement.

[0004] Therefore, how to develop a visual measurement device for passengers' carry-on luggage based on existing technology is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the problems existing in the prior art, a visual measurement method for passenger carry-on baggage at airports is provided. This method involves installing a depth camera at the security checkpoint and measuring the quantity and size of passenger carry-on baggage based on the captured images.

[0006] The technical solution adopted in this invention is as follows: A visual measurement method for airport passenger carry-on baggage, comprising:

[0007] A depth camera is placed above and to the side of the security checkpoint.

[0008] The system uses depth cameras to capture images of passengers and their carry-on luggage, and determines the amount of carry-on luggage based on the captured images.

[0009] A single-view 3D point cloud map is generated from the image, and then the dual-view local point cloud maps are merged and stitched together to generate a 3D point cloud map and then 3D reconstruction is performed.

[0010] Based on the 3D reconstruction model, the model of personal information is projected or separated, and the size of the luggage in the model is calculated.

[0011] The system determines whether the size or quantity of luggage exceeds a preset threshold; if so, it prompts the passenger to take action.

[0012] As a preferred embodiment, determining the quantity of carry-on luggage based on the acquired images specifically includes:

[0013] After segmenting, denoising, and completing the image, classification and recognition are performed. Then, the classified and recognized images are clustered to obtain the target passenger's body and luggage in the image.

[0014] By distinguishing between luggage and human body through a pre-built luggage dataset, the number of luggage items in the two depth camera images is obtained. The number of luggage items in the two depth camera images is added together to obtain the number of carry-on luggage items of the target passenger.

[0015] As a preferred embodiment, obtaining the target passenger's body and luggage in the image specifically includes:

[0016] Semantic segmentation was performed on the acquired images, and MioU was used as the evaluation metric to segment the target passengers in the images;

[0017] The target passenger at the segmentation point is restored to an image, and instance segmentation is performed on the image to segment the target passenger's body and luggage.

[0018] As a preferred embodiment, the specific process of the three-dimensional reconstruction includes:

[0019] Generate a single-view 3D point cloud map from the acquired images, and calibrate and calculate the transformation matrix;

[0020] A transformation matrix is ​​used to unify the side view point cloud map to the front view point cloud map in the same coordinate system to complete the coarse matching of the point cloud;

[0021] The ICP algorithm is used iteratively to complete the fine matching between point clouds and obtain the initial 3D reconstruction model;

[0022] The initial 3D reconstruction model is trained using deep learning algorithms such as DQN. The 3D reconstruction model before training is compared with the measured data. When the error after training is less than 1%, the final 3D reconstruction model is obtained.

[0023] As a preferred embodiment, the calculation of luggage dimensions in the model specifically includes:

[0024] The 3D reconstruction model is divided into regions from top to bottom: 0-10% is region one, 10%-50% is region two, and 50%-100% is region three.

[0025] Calculate luggage dimensions based on model projection measurement points;

[0026] Calculate the length, width, and height dimensions of the luggage based on the measuring points of the separated luggage model;

[0027] Compare the luggage dimensions calculated from the model projection measurement points with the luggage length, width and height dimensions calculated from the separated luggage model measurement points. If the error is less than 1%, either calculation result can be selected as the luggage dimensions.

[0028] As a preferred approach, if the error between the luggage size calculated from the model projection measurement points and the luggage length, width and height size calculated from the separated luggage model measurement points is greater than 5%, then a carry-on luggage database is constructed to perform deep learning on carry-on luggage, search for luggage sizes similar to carry-on luggage for verification, and adjust the model or calculation method to redetermine the luggage size.

[0029] As a preferred embodiment, the calculation of luggage dimensions based on model projection measurement points specifically includes:

[0030] Calculate luggage height based on model side projection measurement points: Project the model onto the xOz plane, and calculate the luggage height based on the three-dimensional coordinates of the two points. The height of the backpack or shoulder bag can be obtained within region two, and the height of the suitcase can be obtained within region three.

[0031] Calculate the width of the luggage based on the top-view projection size of the model: Project the model onto the xOy plane, extract the outline directly within the projection, and the first and second convex points on the outer contour are the vertices of the luggage. Calculate the distance between these two convex points to get the width size, and you can get the width size of the suitcase within the area of ​​region three.

[0032] Calculate the luggage thickness based on the projection measurement points at 45° or other deflection angles of the model: Project the model onto the xOz or yOz plane deflection plane, extract the contour line directly within the projection, and the outermost convex point and inner convex point on the outer contour of region two are the thickness vertices of the luggage. Calculate the distance between these two convex points to obtain the thickness dimension.

[0033] As a preferred option, the depth camera positioned above the security checkpoint is implemented using a binocular camera, a TOF camera, or other types of cameras.

[0034] As a preferred option, the depth camera positioned above the side of the security checkpoint can be either a fixed or target-tracking camera.

[0035] Compared with the existing technology, the beneficial effects of adopting the above technical solution are as follows: the present invention can measure the quantity and size of passengers' carry-on luggage without being noticed during security checks, without affecting the efficiency of passengers' security check passage, and can improve the management level of passengers' "oversized" and "oversized" carry-on luggage, improve the on-time rate of flights, and reduce the operating costs of airlines. Attached Figure Description

[0036] Figure 1 This is a flowchart of the visual measurement method for airport passenger carry-on baggage proposed in this invention.

[0037] Figure 2 This is a schematic diagram of the depth camera arrangement in one embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of luggage quantity detection in one embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram of the region division of a three-dimensional reconstruction model in one embodiment of the present invention. Detailed Implementation

[0040] The embodiments of this application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. Rather, the embodiments of this application include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0041] To achieve the following objectives:

[0042] First, check the number of passengers' carry-on luggage, including backpacks, shoulder bags, handbags, and suitcases;

[0043] Second, detect the three-dimensional dimensions of passengers' backpacks, handbags, suitcases, etc.

[0044] Third, passengers do not need to put down their luggage for contactless measurement.

[0045] This invention proposes a visual measurement method for airport passenger carry-on baggage. Please refer to [link / reference]. Figure 1 The specific plan is as follows:

[0046] Step 1: Place a depth camera above and to the side of the security checkpoint.

[0047] The two depth cameras installed at the security checkpoint form the basis of the entire system. They mainly collect images of passengers and their luggage in real time, and then analyze the images.

[0048] In this embodiment, the depth camera positioned above and in front of the security checkpoint can be a binocular camera, a TOF camera, or other types of cameras. The depth camera positioned above and to the side of the security checkpoint can be a fixed or target-tracking camera. In one embodiment, the depth camera can be implemented using a Microsoft Kinect series depth camera or an Intel RealSense series depth camera. Please refer to... Figure 2 By arranging two cameras, one in front of and one to the side of the passenger, the dual-view depth camera can increase the field of view and cover the outer surface of the passenger and luggage as much as possible.

[0049] In practical applications, guide lines can be set up on the ground at the security checkpoint queue area, along with soft barriers, so that passengers can correctly appear within the shooting range of the two depth cameras.

[0050] Step 2: Use a depth camera to capture images of the passenger and their carry-on luggage, and determine the amount of carry-on luggage based on the captured images.

[0051] In this embodiment, please refer to Figure 3 After the depth camera captures the image, it needs to be segmented, denoised, and completed. Then, it needs to be classified and identified. The classified and identified images are then clustered to separate the target passenger's body and luggage. Finally, luggage and body are distinguished by a pre-built luggage dataset, and the number of luggage in the two depth camera images is obtained. The sum of these two data points gives the number of carry-on luggage items of the target passenger.

[0052] The following explanation uses an image captured by one of the cameras as an example, specifically including:

[0053] (1) First, the acquired images are segmented, mainly using semantic segmentation. Initial training can be performed using the ShapeNet dataset. MioU is used as the evaluation metric to segment the target passengers in the images. MIoU, short for Mean Intersection over Union, is a metric for measuring the performance of semantic segmentation algorithms. Its basic idea is to compare the predicted segmentation results with the actual segmentation results, calculate their intersection and union, and then calculate the ratio of the intersection to the union. Specifically, for a pixel, if it is correctly predicted as a certain category, it is considered to belong to the intersection of the predicted and actual results; if the pixel is incorrectly predicted as another category, it is considered to belong to the union of the predicted and actual results.

[0054] (2) Restore the segmented passenger to an image and perform instance segmentation on the image. When segmenting, you can also use the ShapeNet dataset for initial training to segment the target passenger's body and luggage.

[0055] (3) Then, the luggage and human body are distinguished by the constructed luggage dataset to obtain the number of luggage N1 of the target passenger in the image; In this embodiment, the luggage dataset is a dataset about the vast majority of backpacks, shoulder bags, suitcases, etc., including three-dimensional contours, pixel size, projection size, etc., and the luggage and human body are distinguished by the luggage data in the luggage dataset.

[0056] Repeat the above steps for the image captured by the other camera to obtain the number of bags N2 belonging to the target passenger in that image. Thus, the number of carry-on bags N = N1 + N2 is calculated. If the number of carry-on bags exceeds n pieces, the passenger is advised to consolidate the bags or check them in separately. Here, n is set according to the regulations of the Civil Aviation Administration and airlines. This embodiment also includes deduplication processing for baggage numbers; that is, if the same piece of baggage appears in the images captured by both cameras, it is counted as only one piece of baggage in the calculation.

[0057] Step 3: Generate single-view 3D point cloud maps based on the images, then merge and stitch the dual-view local point cloud maps to generate a 3D point cloud map and perform 3D reconstruction.

[0058] Specifically, a single-view 3D point cloud map is generated based on the acquired images, and the transformation matrix T is calibrated and calculated;

[0059]

[0060] Where R is the optimal rotation matrix and t is the translation matrix.

[0061] R = VU T

[0062] Where V and U are orthogonal matrices of the singular value decomposition.

[0063] t=-R×C A +C B

[0064] Where C A C B These are the center points of the sphere center coordinate sets A and B within the common field of view of the depth camera, respectively.

[0065] Secondly, the transformation matrix is ​​used to unify the side view point cloud map to the front view point cloud map in the same coordinate system to complete the coarse matching of the point cloud;

[0066] Next, the ICP algorithm is used iteratively to complete the fine matching between point clouds and obtain the initial 3D reconstruction model;

[0067] Finally, the initial 3D reconstruction model is trained using deep learning algorithms such as DQN. The 3D reconstruction model before training is compared with the measured data. When the error after training is less than 1%, the final 3D reconstruction model is obtained and put into use.

[0068] Step 4: Based on the 3D reconstruction model, project or separate the model of personal information and calculate the luggage size in the model.

[0069] Please refer to the following before calculating luggage dimensions. Figure 4First, the 3D reconstruction model needs to be divided into regions, from top to bottom: 0-10% is region one (head), 10%-50% is region two (torso), and 50%-100% is region three (below the thigh).

[0070] In this embodiment, two methods for calculating and verifying luggage dimensions are proposed.

[0071] First, calculate the luggage dimensions based on the model's projected measurement points, including:

[0072] (1) Calculate the height of the luggage based on the side projection measurement points of the model:

[0073] Projecting the model onto the xOz plane, the height data can be obtained from two points (x, y, z). i y i , z i ) and (x j y j , z j The three-dimensional coordinates of the object are calculated:

[0074]

[0075] This allows us to obtain the height dimensions of a typical backpack or shoulder bag within area two, and the height dimensions of a rolling suitcase within area three.

[0076] (2) Calculate the width of the luggage based on the top-view projection dimensions of the model:

[0077] Project the model onto the xOy plane and extract the contour lines directly within the projection. The first and second convex points on the outer contour are the vertices of the luggage. Calculate the distance between these two convex points to obtain the width dimension. Thus, the width dimension of the suitcase can be obtained within region three.

[0078] (3) Calculate the luggage thickness based on the projection measurement points at 45° or other deflection angles of the model:

[0079] Project the model onto the xOz or yOz plane deflection plane, and directly extract the contour line within the projection. The outermost convex point and inner convex point on the outer contour of region two are the thickness vertices of the luggage. Calculate the distance between these two convex points to obtain the thickness dimension.

[0080] Then, the length, width, and height dimensions of the luggage are calculated based on the measurement points of the separated luggage model. In this embodiment, the separated luggage model can be calculated based on pixel size or by projecting it separately.

[0081] Finally, the luggage dimensions calculated from the model's projected measurement points are compared with those calculated from the separated luggage model's measurement points. If the error is less than 1%, either calculation result is selected as the luggage dimension. However, if the error between the luggage dimensions calculated from the model's projected measurement points and those calculated from the separated luggage model's measurement points exceeds 5%, a carry-on luggage database is constructed for deep learning of carry-on luggage. Similar luggage dimensions are then searched for for verification, and the model and calculation method are adjusted to re-determine the luggage dimensions.

[0082] Step 5: Determine if the size or quantity of luggage exceeds a preset threshold. If it does, prompt the passenger to take action.

[0083] The system compares the calculated baggage dimensions with the dimensions set by the airline, and the calculated baggage quantity with the quantity set by the airline. If the baggage exceeds a certain threshold, the passenger will be reminded that the baggage exceeds the limit and needs to be checked in.

[0084] In particular, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts.

[0085] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0087] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0088] In another aspect, this application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the airport passenger carry-on baggage visual measurement method described in the above embodiments.

[0089] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the airport passenger carry-on baggage visual measurement method described in the above embodiments.

[0090] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0091] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0092] For those skilled in the art, the specific meanings of the above terms in this invention can be understood according to the specific circumstances; the accompanying drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of this invention. Obviously, the described embodiments are some embodiments of this invention, but not all embodiments. Generally, the components of the embodiments of this invention described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0093] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for visually measuring carry-on baggage of airport passengers, characterized in that, include: A depth camera is placed above and to the side of the security checkpoint. The system uses depth cameras to capture images of passengers and their carry-on luggage, and determines the amount of carry-on luggage based on the captured images. A single-view 3D point cloud map is generated from the image, and then the dual-view local point cloud maps are merged and stitched together to generate a 3D point cloud map and then 3D reconstruction is performed. Based on the 3D reconstruction model, the model of carry-on luggage is projected or separated, and the size of the luggage in the model is calculated. The system determines whether the size or quantity of luggage exceeds a preset threshold; if so, it prompts the passenger to take action. The specific process of the three-dimensional reconstruction includes: Generate a single-view 3D point cloud map from the acquired images, and calibrate and calculate the transformation matrix; A transformation matrix is ​​used to unify the side view point cloud map to the front view point cloud map in the same coordinate system to complete the coarse matching of the point cloud; The ICP algorithm is used iteratively to complete the fine matching between point clouds and obtain the initial 3D reconstruction model; The initial 3D reconstruction model is trained using the DQN deep learning algorithm. The trained 3D reconstruction model is then compared with the measured data. When the training error is less than 1%, the final 3D reconstruction model is obtained. The calculated luggage dimensions in the model specifically include: The 3D reconstruction model is divided into regions from top to bottom: 0-10% is region one, 10%-50% is region two, and 50%-100% is region three. Calculate luggage dimensions based on model projection measurement points; Calculate the length, width, and height dimensions of the luggage based on the measuring points of the separated luggage model; Compare the luggage dimensions calculated from the model projection measurement points with the luggage length, width and height dimensions calculated from the separated luggage model measurement points. If the error is less than 1%, either calculation result can be selected as the luggage dimensions. The calculation of luggage dimensions based on model projection measurement points specifically includes: Calculate luggage height based on model side projection measurement points: Project the model onto the xOz plane, and calculate the luggage height based on the three-dimensional coordinates of the two points. The height of the backpack or shoulder bag can be obtained in area two, and the height of the suitcase can be obtained in area three. Calculate the width of the luggage based on the top-view projection size of the model: Project the model onto the xOy plane, extract the outline directly within the projection, and the first and second convex points on the outer contour are the vertices of the luggage. Calculate the distance between these two convex points to get the width size, and you can get the width size of the suitcase within the area of ​​region three. Calculate the luggage thickness based on the projection measurement points at 45° or other deflection angles of the model: Project the model onto the deflection plane of the xOz or yOz plane, extract the contour line directly within the projection, and the outermost convex point and inner convex point on the outer contour of region two are the thickness vertices of the luggage. Calculate the distance between these two convex points to obtain the thickness dimension.

2. The method for visually measuring airport passenger carry-on baggage according to claim 1, characterized in that, The step of determining the quantity of carry-on luggage based on the collected images specifically includes: After segmenting, denoising, and completing the image, classification and recognition are performed. Then, the classified and recognized images are clustered to obtain the target passenger's body and luggage in the image. By distinguishing between luggage and human body through a pre-built luggage dataset, the luggage in the two depth camera images is obtained. The luggage in the two depth camera images is added together to obtain the number of carry-on luggage of the target passenger.

3. The method for visually measuring airport passenger carry-on baggage according to claim 2, characterized in that, The obtained image of the target passenger's body and luggage specifically includes: Semantic segmentation was performed on the acquired images, and MIoU was used as the evaluation metric to segment the target passengers in the images; The segmented target passenger is restored to an image, and instance segmentation is performed on the image to segment the target passenger's body and luggage.

4. The method for visually measuring airport passenger carry-on baggage according to claim 1, characterized in that, If the error between the luggage size calculated from the model projection measurement points and the luggage length, width and height size calculated from the separated luggage model measurement points is greater than 5%, then a carry-on luggage database is constructed to perform deep learning on carry-on luggage, search for luggage sizes similar to carry-on luggage for verification, and adjust the model or calculation method to redetermine the luggage size.

5. The method for visually measuring airport passenger carry-on baggage according to claim 1, characterized in that, The depth camera positioned above the security checkpoint is implemented using a binocular camera, a TOF camera, or other types of depth cameras.

6. The method for visually measuring airport passenger carry-on baggage according to claim 1, characterized in that, The depth camera positioned above and to the side of the security checkpoint is either a fixed camera or a target-tracking camera.

Images