A terminal luggage transportation whole-process tracking and identification method based on machine vision

Abstract

The present application is directed to the problem that the luggage in the terminal cannot be tracked and identified in the whole process due to the lack of information in the transportation process, and provides a luggage transportation whole process tracking and identification method based on machine vision in the terminal, which mainly relates to the field of image processing. First, a number of checkpoints and their matching pools are set on the luggage conveyor belt; second, the one-way sequential transmission property of the conveyor belt and the matching pool of the checkpoints are used to limit the matching amount of luggage tracking and identification; then, the update of the luggage image and the transmission of the luggage information between the checkpoints are used to realize the whole process tracking and identification of the luggage transportation; in addition, the specific operation of tracking and identification of the luggage in the merging, merging and comprehensive transportation is given; finally, the possible transportation abnormalities are found by checking the matching pool and the abnormal luggage is located. This method better solves the lack of information in the transportation process of the terminal luggage, and further realizes the whole process tracking and identification, and has wide applicability.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology, and in particular to a terminal building luggage transportation whole-process tracking and identification method based on machine vision. BACKGROUND

[0002] The smart airport is one of the core elements of the smart civil aviation construction, and the luggage check-in is an important function of the airport. The upgrading and reconstruction of the luggage check-in system for passengers is the front direction of the intelligent and digital airport in recent years. The luggage check-in steps mainly include luggage transportation in the terminal building, sorting, loading, loading, unloading, luggage on the turntable, and passenger pick-up, etc. However, for most airports, due to the long distance of the luggage transportation in the terminal building, the traditional tracking and identification method cannot cover the whole process of the luggage transportation in the terminal building, and the airport managers cannot real-time view the transportation information of the luggage in the terminal building, which may lead to luggage loss, transportation delay, and further impact on the flight time. Therefore, it is of great significance to explore an effective whole-process tracking and identification method for the luggage transportation in the terminal building for the construction of the smart airport and the improvement of the passenger travel experience.

[0003] Currently, the tracking and identification of the luggage transportation in the terminal building mainly uses the radio frequency identification (RFID) technology. Specifically, an entity electronic tag with passenger information is attached to each piece of luggage, and the node transportation information of the luggage is obtained by manually scanning the electronic tag at specific nodes through the terminal device. However, due to the cost and other problems, it is difficult to densely set scanning nodes on the several kilometers or even tens of kilometers long luggage conveyor belt in the terminal building. The few nodes result in a long distance between each node, which leads to a slow update of the transportation information of the luggage in the terminal building, and the latest position of the luggage cannot be found in time. Moreover, the position and state of the luggage between the nodes cannot be known, and if the luggage transportation is abnormal during this period, the problem and the abnormal luggage cannot be found in time due to the lack of information. Therefore, the RFID identification technology with large nodes cannot achieve the whole-process tracking and identification of the luggage transportation in the terminal building. Compared with the RFID method, the machine vision method can set more nodes on the conveyor belt, track and identify the luggage by using the image recognition technology, find the position of the corresponding luggage at any time through the passenger information, and obtain the visual image of the luggage to achieve the whole-process tracking and identification. However, due to the large number of luggage and the high similarity of the shape and color, it is a difficult problem to solve the tracking and identification of the luggage by using the visual solution. SUMMARY

[0004] The purpose of the present application is to solve the problem that the luggage in the terminal cannot be tracked and identified in the whole process due to the lack of information in the transportation process. In order to achieve this purpose, the present application provides a terminal luggage transportation whole-process tracking and identification method based on machine vision, which is characterized by limiting the matching amount of luggage tracking and identification, realizing real-time supervision of luggage transportation, and positioning the abnormal luggage position. The present application mainly includes five parts: the first part is the setting of the checkpoint and the matching pool; the second part is the binding of the luggage and passenger information; the third part is the general transmission of the luggage on the conveyor belt; the fourth part is the in-out and comprehensive transportation of the luggage on the conveyor belt; and the fifth part is to check whether the luggage transportation is abnormal.

[0005] The first part includes three steps:

[0006] Step 1: According to the specific length of the luggage transportation conveyor belt in the terminal and the transportation speed of the conveyor belt, the number of appropriate checkpoint erection positions is selected, and the starting position is the check-in point of the terminal, and the terminal position is the end of the luggage conveyor belt;

[0007] Step 2: A simple gantry is installed at each checkpoint, and a camera for collecting visual image information is hung on the gantry;

[0008] Step 3: A matching pool is set for all checkpoints in the system to store the luggage images collected by the checkpoint.

[0009] The second part includes two steps:

[0010] Step 4: The passenger registers information at the check-in point, submits the luggage to be shipped, and the luggage is uploaded to the conveyor belt;

[0011] Step 5: The luggage image taken by the camera at the check-in point, that is, the first checkpoint, is bound with the passenger information corresponding to the luggage in step 4.

[0012] The third part includes three steps:

[0013] Step 6: During transportation, the luggage image of the luggage passing through the checkpoint is stored in the matching pool;

[0014] Step 7: The luggage image is tracked and identified by using the checkpoint matching pool and image re-identification technology, and other luggage images corresponding to the same luggage are found; the re-identified matching object is limited to the luggage image in the matching pool of the previous checkpoint by using the one-way sequential transmission property of the conveyor belt; after the matching is completed, the luggage image of the luggage in the matching pool of the previous checkpoint is deleted;

[0015] Step 8, after the matching in step 7 is completed, the luggage is considered to be successfully transmitted to the next checkpoint, and the luggage information bound by the luggage image in step 5 is also transmitted between the checkpoints, thereby realizing real-time supervision during luggage transportation.

[0016] The fourth part includes three steps:

[0017] Step 9, tracking and identifying the luggage in the case of in-bound transmission;

[0018] Step 10, tracking and identifying the luggage in the case of out-bound transmission;

[0019] Step 11, tracking and identifying the luggage in the case of comprehensive transmission.

[0020] The fifth part includes two steps:

[0021] Step 12, checking whether the luggage transportation is normal by matching the pool;

[0022] Step 13, if an exception occurs, locking the position of the abnormal luggage according to the position of the exception matching pool, and solving the abnormal problem.

[0023] The application provides a method for tracking and identifying the whole process of luggage transportation in a terminal building based on machine vision. Firstly, a plurality of checkpoints and corresponding matching pools are arranged on a luggage conveyor belt. Secondly, the matching amount of luggage tracking and identification is limited by the one-way sequential transmission property of the conveyor belt and the checkpoints. Thirdly, the whole process of luggage tracking and identification is realized by updating the luggage image and transmitting the luggage information between the checkpoints. In addition, the specific operation of luggage tracking and identification in the cases of in-bound, out-bound and comprehensive transportation is given. Finally, possible transportation abnormalities can be found by checking the matching pool, and the abnormal luggage can be located. The method solves the problem of information loss in the traditional luggage tracking and identification method, realizes the whole process of luggage tracking and identification in the terminal building, and has wide applicability. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The checkpoint of the application is built;

[0025] Figure 2-1 、 Figure 2-2 、 Figure 2-3 The luggage tracking and identification diagram of the application in the general transmission case is shown;

[0026] Figure 3 The luggage tracking and identification diagram of the application in the in-bound transmission case is shown;

[0027] Figure 4 The luggage tracking and identification diagram of the application in the out-bound transmission case is shown;

[0028] Figure 5 This is a diagram illustrating baggage tracking and identification under integrated transmission conditions, based on the present invention. Implementation

[0029] To better understand this invention, the machine vision-based end-to-end tracking and identification method for airport baggage transportation is described in more detail below with reference to specific embodiments. In the following description, detailed descriptions of existing technologies may obscure the subject matter of this invention, and these descriptions will be omitted here.

[0030] Step 1: Select the appropriate number of checkpoint locations based on the specific length and speed of the baggage conveyor belt in the terminal. The starting point, i.e., the first checkpoint, should be located at the terminal check-in point, and the ending point, i.e., the last checkpoint, should be located at the end of the terminal baggage conveyor belt. The distance between each checkpoint should allow sufficient space for baggage tracking and identification, while also ensuring that the speed of updating transportation information meets the real-time requirements of the entire process.

[0031] Step 2: Install a simple gantry at each checkpoint, and suspend the camera that collects visual image information on the gantry, ensuring its field of view is perpendicular to the conveyor belt. The setup should look like this. Figure 1 As shown; the field of view captured by the camera mainly includes the conveyor belt and the luggage on the conveyor belt, excluding the surrounding environment; if the ambient light around the checkpoint is insufficient, lighting is installed on the gantry; the purpose of this step is to obtain high-quality luggage images for subsequent tracking and identification;

[0032] Step 3: Set up a matching pool for storing baggage images in the system for each checkpoint. Whenever baggage passes through a checkpoint, the checkpoint camera must take an image of the current baggage, number the image, and put it into the matching pool of that checkpoint. This step is performed for all checkpoints.

[0033] Step 4: Passengers register their information at the check-in counter, complete the baggage check-in procedures, and submit the baggage to be checked in; staff place the checked baggage on the conveyor belt, ensuring that the maximum contact surface of the baggage is parallel to the conveyor belt, so as to obtain baggage image information to the greatest extent.

[0034] Step 5: When luggage passes through the first checkpoint at the check-in location, the camera takes pictures of the luggage passing through the checkpoint in turn, collecting the current luggage image information. However, in machine vision, it is not possible to know which passenger the luggage belongs to based solely on the luggage image. Therefore, it is necessary to bind the luggage image information collected in this step with the passenger information corresponding to the luggage in Step 4 in the system. Subsequent checkpoints obtain the corresponding passenger information based on the recognized luggage image, thereby determining which passenger the luggage belongs to. See Step 8 for details.

[0035] Figure 2-1 、 Figure 2-2 、 Figure 2-3 is the luggage tracking identification diagram of the present application in the general transmission case, all the Roman numerals in the diagram are only for distinguishing different pieces of luggage and luggage images, and do not represent absolute sequence, in the present embodiment, the tracking identification is specifically performed according to the following steps:

[0036] Step 6, Figure 2-1 is the n, n+1, n+2 checkpoints on the conveyor belt, which is used as a general example to illustrate the logic of luggage tracking identification of the present application; according to the requirement of step 3, the camera of the n checkpoint has collected the image information of the luggage I, II, III passing through the checkpoint, which are numbered as luggage image I, luggage image II, and luggage image III, and the three collected luggage images are put into the matching pool n.

[0037] Step 7, with the transportation of the luggage, the subsequent luggage will pass through the n checkpoint in succession, and the collected luggage images are also continuously stored in the matching pool n, when the luggage I passes through the n+1 checkpoint, the camera of the n+1 checkpoint also takes a photo of the current passing luggage I, and stores the collected luggage image V into the matching pool n+1, as shown in Figure 2-2 The tracking identification of the luggage is specifically implemented as follows:

[0038] Step 7-1, the image re-identification technology is used to track and identify the luggage image V, and the purpose of the matching is to find the luggage image of the same piece of luggage corresponding to the luggage image V; but according to the general re-identification idea, the matching object should be all the luggage images collected at the check-in point, that is, the first checkpoint, but in the total amount of all the luggage images, due to the large number of luggage, the high similarity of the color and appearance of the luggage, the difficulty of re-identification will be very great; and one of the features of the present application is to set several checkpoints on the conveyor belt, which is equivalent to dividing the total mileage of the conveyor belt into several small areas, and due to the nature of the one-way sequential transmission of the conveyor belt, the luggage I just passing through the n+1 checkpoint has not passed through the n+2 checkpoint, so the image of the same piece of luggage will only appear in the matching pool of the checkpoint it has passed through, and it is impossible to have its image in the matching pool of the checkpoint it has not passed through, therefore, in the method of the present application, the tracking identification of a piece of luggage no longer needs to be matched with the total amount of all the luggage images, but the matching object is limited to the luggage images in the matching pool of the checkpoint it has passed through, and further, is directly limited to the luggage images in the matching pool of the adjacent previous checkpoint, which greatly eliminates the redundant luggage images without matching value, and embodies the feature of the present application of limiting the matching amount of luggage tracking identification by using the checkpoint matching pool.

[0039] Step 7-2, according to step 7-1, the luggage image V in the matching pool n+1 is matched with all the luggage images in the matching pool n; in this way, each time the luggage image only needs to be matched with the matching pool of the last checkpoint; it is emphasized that although the luggage images are stored in the matching pool in sequence, due to the possibility of merging of the conveyor belt between the two checkpoints during the luggage transportation, the first luggage passing through the last checkpoint may not be the first one passing through the next checkpoint, which means that the number and sequence of the luggage images stored in the matching pool may not be completely corresponding to the number and sequence of the luggage images stored in the last matching pool, so this step is to re-identify and match the luggage image V in the matching pool n+1 with all the luggage images in the matching pool n, instead of directly matching with the first luggage image I stored in the matching pool n; meanwhile, due to the influence of the transmission speed of the conveyor belt, the number of luggage in the matching pool n may not be the same as the number in the matching pool n+1, if the number is not zero, it is executed from step 7-1, if the number is zero, it needs to wait until there is luggage passing through to execute from step 7-1. Figure 2-2

[0040] Step 7-3, as the luggage transportation proceeds, the number of luggage images in each checkpoint matching pool will accumulate more and more, which will lead to the matching amount increasing more and more, until the number tends to the total amount of luggage; after the matching is successful, the luggage I will continue to transport to the checkpoint n+2, when it passes through the checkpoint n+2 later, the matching object changes from the matching pool n to the matching pool n+1, then the luggage image I in the matching pool n becomes a redundant image without matching value in the subsequent matching of the luggage, so after the luggage image V is matched with the luggage image I corresponding to the same luggage I, the luggage image I is deleted, and the luggage image V becomes the latest image of the luggage, as shown in Figure 2-3 ; in this way, after each matching is successful, the luggage image stored in the last matching pool needs to be deleted, which makes the matching amount of each luggage each time maintain a small scale, that is, during the luggage transportation, each checkpoint matching pool only contains the luggage image of the luggage that has passed through the checkpoint but has not passed through the next checkpoint, which further embodies the feature of the present application that the checkpoint matching pool is used to limit the matching amount of luggage tracking and identification; it is emphasized that when there are two or more luggage with the same color and appearance on the conveyor belt, it is possible that two or more almost completely consistent luggage images appear in a matching pool at the same time, so if multiple images are matched successfully in the matching, according to the nature of the one-way sequential transmission of the conveyor belt, the one that is first stored in the matching pool can be deleted after the matching is successful.

[0041] ​Step 8, after the luggage image V and the luggage image I in step 7-2 are matched to correspond to the same luggage I, it is considered that the luggage I has been successfully transported from the checkpoint n to the checkpoint n+1; in this way, after each successful matching, it is considered that a piece of luggage has been successfully transported from a checkpoint to the next checkpoint, and the luggage image of the luggage is updated; however, if only the luggage image is obtained, it cannot be known which passenger's luggage it is, and the passenger information corresponding to the luggage I has been bound to the luggage image since the first checkpoint, and in each successful re-identification matching, the luggage image left by the luggage I at a checkpoint is transmitted to the luggage image left at the next checkpoint, thereby realizing the transmission of the passenger information corresponding to the luggage I between the entire conveyor belt checkpoints, and the large number of checkpoints on the conveyor belt ensures a high information update frequency, so that after the image of the luggage is matched, it can be known which passenger's luggage it is, that is, the latest position of the luggage of the passenger can be inquired at any time, and the latest image of the luggage can also be obtained to master the luggage transportation state, thereby embodying the real-time supervision feature of the luggage transportation in the terminal building.

[0042] Due to the very complex actual structure of the terminal building luggage transportation system, in addition to the general transportation situation described above, the conveyor belt may also have situations such as Figure 3 、 Figure 4 、 Figure 5 inflow, outflow, and comprehensive transportation, and the tracking and identification of the luggage according to the present application are performed according to the following steps when encountering these situations.

[0043] Step 9, in the inflow scene as shown in Figure 3 , the n th and n+1 th checkpoints must be arranged on both sides of the inflow port, that is, the inflow port is contained between the two checkpoints, and a checkpoint na1 is arranged at the inflow port, where a represents inflow, and na1 represents the first inflow checkpoint between the checkpoints n and n+1, at this time, the checkpoint na1 is equivalent to the end point of the branch, and all the luggage images successively stored in the matching pool na1 are added to the luggage images in the matching pool n, which are combined to become the matching objects required by the checkpoint n+1, and after the luggage images in the matching pool na1 are added to the matching pool n, the corresponding luggage images stored in the matching pool na1 are deleted, at this time, the main road and the branch are combined into one in the concept of the matching pool, and the inflow situation is changed into the general transportation situation, which is Figure 2-1 .

[0044] Step 10, in the outflow scene as shown in Figure 4In the shown export scene, the nth and the n+1th checkpoints must also be set on both sides of the export port, and a checkpoint nb1 is set at the export port, where b represents export, and nb1 represents the first export checkpoint between the checkpoints n and n+1; when luggage passes through the checkpoint nb1, the luggage image is first matched with the matching pool n, and after a successful match, the corresponding luggage image in the matching pool n is deleted, and the deleted matching pool n becomes the matching object required for the checkpoint n+1, which is equivalent to dividing the main road into two, stripping the branched road, and transforming into Figure 2-1 the case; and the export port is equivalent to the starting point of a new route, and the subsequent route is also consistent with Figure 2-1 the case.

[0045] Step 11, for the comprehensive case of both import and export, its essence is to perform the corresponding step 9 or step 10 operation on each import or export according to the transport direction; as shown in Figure 5 , it contains an import port and an export port, and the conveyor belt transports from left to right, so the first import point on the left forms the Figure 3 import case with the main road, step 9 is performed, and after step 9 is completed, the two roads are conceptually combined into a new main road, which at this time forms the Figure 4 export case with the second export point, step 10 is performed, and after step 10 forms a new main road, it forms the Figure 3 or Figure 4 case with the next import or export, and the corresponding steps are executed according to the sequence until all import and export operations are completed.

[0046] Step 12, judge whether the luggage transportation is normal, according to the above description, when the luggage passes through a checkpoint, the luggage image collected is stored in the matching pool of the checkpoint, and after the luggage image is successfully matched with the luggage image in the matching pool of the subsequent checkpoint, it is deleted, so as to achieve dynamic balance of the number of luggage images in the matching pool during luggage transportation, therefore, if the luggage image in the matching pool except the last checkpoint remains for too long, it can be judged that the luggage has not reached the next checkpoint, which is considered as an abnormal case; and as the luggage gradually completes transportation, the matching pool of all checkpoints except the last one gradually no longer has new luggage images stored, the number of luggage images in these matching pools will gradually decrease until it is deleted to zero, when the luggage transportation is completely finished, the number of luggage in the last matching pool should be the total amount of luggage, if there are still matching pools remaining luggage images except the last matching pool, it is also considered as an abnormal case;

[0047] If the matching pool appears the abnormal condition described in step 12, it indicates that the luggage image corresponding to the luggage has successfully passed the current checkpoint, but due to the abnormal condition such as being stuck in the conveyor belt somewhere or falling out of the conveyor belt, the luggage has not passed the next checkpoint, so that the luggage image has not been deleted for a long time during transportation or even still exists at the end of transportation, at this time, the position of the abnormal luggage can be quickly locked between the checkpoint corresponding to the matching pool where the abnormality occurs and the next checkpoint, then the airport manager can send staff to the area to handle the abnormal luggage, and the transportation of the luggage is restored to normal, which embodies the feature of the application that the abnormal luggage is positioned using the checkpoint matching pool.

[0048] The application provides an airport terminal luggage transportation whole-process tracking and identification method based on machine vision, first, a plurality of checkpoints and corresponding matching pools are arranged on the luggage conveyor belt, second, the matching amount of luggage tracking and identification is limited by using the one-way sequential transmission property of the conveyor belt and the checkpoint matching pool, then, the whole-process tracking and identification of the luggage transportation is realized by using the update of the luggage image between the checkpoints and the transmission of the luggage information, in addition, the specific operation of the luggage tracking and identification under the conditions of luggage in, out and comprehensive transportation is given, finally, the possible transportation abnormality is found by checking the matching pool, and the abnormal luggage is positioned.

[0049] Although the above describes the specific embodiments of the application, it should be clear that the application is not limited to the scope of the specific embodiments, and for those skilled in the art, all the application creations using the concept of the application are within the protection scope as long as the various changes are obvious within the spirit and scope of the application defined and determined by the appended claims.

Claims

1. A method for tracking and identifying a whole process of luggage transportation in a terminal based on machine vision, characterized in that, The application discloses a method for tracking and identifying luggage in the whole process of luggage transportation by using machine vision, which comprises five parts: setting of check points and matching pools, binding of luggage and passenger information, general transmission of luggage on a conveying belt, in-out and comprehensive transmission of luggage on the conveying belt, and checking of abnormal conditions in the luggage transportation. The first part comprises three steps. Step 1: selecting a proper number of check point setting positions according to the specific length of the luggage conveying belt in the terminal and the conveying speed of the conveying belt, with the starting position being the check-in point of the terminal and the ending position being the end of the luggage conveying belt in the terminal; Step 2: installing a simple gantry at each check point, and hanging a camera for collecting visual image information on the gantry; the visual field collected by the camera includes the conveying belt and the luggage on the conveying belt, and the surrounding irrelevant environment is not included; Step 3: setting a matching pool for each check point in the system to store the luggage images collected by the check point; the camera of the check point takes an image of the current luggage when the luggage passes through the check point, and the image is stored in the matching pool of the check point; all the check points perform this step. The second part comprises two steps. Step 4: registering information at the check-in point, submitting the luggage to be transported, and conveying the luggage; Step 5: when the luggage passes through the first check point located at the check-in position, the camera takes a picture of the luggage passing through the check point, and the luggage image is bound with the passenger information corresponding to the luggage in step 4 in the system. The third part comprises three steps. Step 6: collecting the image of the luggage when the luggage passes through a check point during the transportation and storing the image in the matching pool; Step 7: tracking and identifying the luggage image collected in step 6 by using the check point matching pool and image re-identification technology, finding other luggage images of the same luggage corresponding to the luggage image, and the specific implementation is as follows: Step 7-1: limiting the re-identified matching object to the luggage images in the matching pool of the previous check point by using the one-way and sequential transmission property of the conveying belt; Step 7-2: matching the luggage image with all the luggage images in the matching pool of the previous check point; Step 7-3: deleting the luggage image of the luggage in the previous matching pool after the matching is completed, so that the matching amount of each luggage is maintained at a small scale; Step 8: after the matching in step 7 is completed, it is considered that the luggage is successfully transmitted to the next check point, and the luggage information bound with the luggage image in step 5 is also transmitted between the check points, so that it can be known that the luggage belongs to which passenger after the image of the luggage is matched, and the latest position of the luggage of the passenger can be inquired at any time, and the latest image of the luggage can also be obtained to master the transportation state of the luggage, thereby realizing real-time supervision of the luggage during the transportation; The fourth part comprises three steps. Step 9: in the in-out transmission condition, the luggage image in the in-out matching pool is combined with the luggage image in the main road matching pool to form the matching object of the next check point. Step 10, in the case of export transmission, track the luggage, subtract the image of the export matching pool from the image of the main road matching pool, and the remaining image of the main road matching pool is the matching object of the next checkpoint; Step 11, in the case of comprehensive transmission, track the luggage, and perform corresponding steps 9 or 10 operations on each import or export according to the transportation direction; The fifth part includes two steps: Step 12, check the luggage transportation by matching pool, if there is luggage image remaining for too long in the matching pool except the last one during transportation, or if there is luggage image remaining in the matching pool except the last one after transportation, it means that there is an abnormality; Step 13, if the matching pool has the abnormality described in step 12, it means that the luggage image corresponding to the luggage has successfully passed the current checkpoint, but it has not passed the next checkpoint because it is stuck somewhere on the conveyor belt or has fallen out of the conveyor belt, at this time, the position of the abnormal luggage can be quickly locked between the checkpoint corresponding to the matching pool where the abnormality occurs and the next checkpoint, and then the abnormal problem can be solved.

2. The machine vision-based tracking and identification method for the whole process of luggage transportation in the terminal according to claim 1, characterized in that, In step 7, the one-way sequential transmission property of the conveyor belt and the matching pool of the checkpoint are used to limit the matching amount of luggage tracking identification.

3. The machine vision-based tracking and identification method of the whole process of luggage transportation in the terminal according to claim 1, characterized in that, In step 8, the update of the luggage image and the transmission of the luggage information by the checkpoint matching pool are used to realize real-time supervision of the luggage transportation.

4. The machine vision-based tracking and identification method for the whole process of luggage transportation in the terminal according to claim 1, characterized in that, In step 13, the checkpoint matching pool is used to locate the abnormal luggage.

Images