Cross-modal parcel tracking matching method and system based on three-camera X-ray

Abstract

The invention relates to a cross-modal parcel tracking and matching method and system based on three-camera X-ray, and belongs to the field of security check, and the method comprises the steps: when parcel features and human body features under visible light are captured through a front-facing camera, tracking trajectory information variables are generated according to the parcel features, and corresponding parcels are tracked; the human body features are bound with the tracking trajectory information variables; the package features comprise appearance features and motion features; matching is carried out according to the package features under the X-rays and the package features under the visible light in the tracking trajectory information variables, and the package features under the X-rays are updated to the successfully matched tracking trajectory information variables; when new visible light package features and human body features are captured through a rear camera, matching is carried out according to the new package features under visible light and tracking trajectory information variables; and updating the new visible light package features to the successfully matched tracking trajectory information variables, and carrying out second binding on the new human body features and the tracking trajectory information variables.

Description

Technical Field

[0001] The present invention relates to a method and system for tracking and matching a package across modalities based on three-camera X-rays, and belongs to the field of security inspection. Background Art

[0002] In the field of security inspection, information-based tracing systems are increasingly being highly valued. Through innovative technical means, the X-ray bag images of passengers can be quickly matched to the visible light images captured by passengers during the bag placement process, thereby further obtaining video information of the passengers placing their bags, realizing fast and efficient information retrieval and tracking, and solving the problem of consuming a lot of manpower and material resources in the process of security inspection information tracing. The current common security inspection system uses a relatively simple and rigid method to match visible light images with X-ray images, mainly through time intervals and the outline or length of the package. This method performs poorly in some cases: (1) When the packages are of similar size (for example, at some train stations, the outline and length of suitcases are sometimes close), the error of matching by the outline or length of the package is relatively large; (2) When the package is light (for example, at some subway stations, passengers may only carry some small items in a bag), the package may be blocked by the belt at the entrance of the security inspection machine and cannot enter the interior, which will make the time interval method ineffective; (3) When there are a large number of packages and some of them overlap, the outline or time interval of the package alone is not enough to accurately match. The main reason for the poor performance in the above scenarios is the neglect of a large amount of useful information contained in visible light images and X-ray images.

[0003] In the security inspection process, the association of multiple target objects is not detailed enough, and no closed loop of association is formed, and the system's error correction ability is not strong. Specifically, most security inspection processes are now equipped with only front cameras (Front cameras) and X-ray imaging (X-ray cameras), which results in only one round of matching between visible light images and X-ray images (i.e., matching between Front packages and X-ray packages). If the second round of matching fails, the entire process will fail, and the association between people and packages will also fail. The current security inspection system also has a weak traceability capability. The current association process is to associate pedestrians (pedestrians captured by the Front camera) with Front packages, and then associate Front packages with X-ray packages. This model cannot cope with some situations: (1) When the bag loses tracking after leaving the security inspection machine, if the passenger later finds that he has taken the wrong bag, it is difficult to locate who has taken the wrong bag; (2) If the dangerous goods are not identified, it is impossible to trace who the dangerous goods belong to when an adverse event occurs later.

[0004] The Chinese patent document with the announcement number CN113759431B discloses a security inspection data association method and device, and an X-ray security inspection system. The security inspection data association method of the invention includes: extracting a human body image and a package appearance image from the entrance video collected at the entrance of the X-ray security inspection device, and associating the human body image and the package appearance image of the same package; tracking each package in the entrance video to obtain the movement trajectory of each package; associating the package appearance image and the X-ray image of the same package according to the movement trajectory; and associating the human body image, the package appearance image and the X-ray image corresponding to the same package through the package appearance image. In addition, the X-ray security inspection system may also include: an exit image acquisition device, which is arranged at the exit of the X-ray machine, and acquires images containing human bodies and packages. When the human-package association data at the entrance and the human-package association data at the exit are inconsistent, an alarm is issued, thereby being able to detect the situation where the passerby takes the wrong or missed package at the exit. However, when performing image association based on the motion trajectory, the time and position of the X-ray image acquisition are predicted based on the moment when the package enters the X-ray security inspection device and the conveyor belt speed, and then the corresponding X-ray image of the package is extracted for association based on the prediction result. When the package is blocked by the security entrance belt or other obstacles, the package will not arrive at the predicted position at the predicted time, and the application of this method will result in association errors or failure to associate; when there are many packages, a large number of packages will enter the security inspection machine at a similar time, and there may even be overlapping packages. When this method is applied, the same or similar prediction results may appear, making it difficult to associate the images of the packages. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for cross-modal package tracking and matching based on three-camera X-ray, so as to solve the problem that it is difficult to associate package images when the package is blocked by obstacles and delayed in entering or leaving the security inspection machine during security inspection.

[0006] To achieve the above object, the solution of the present invention includes: A method for tracking and matching a package across modalities based on three-camera X-rays of the present invention comprises the following steps: 1) When the package features and human features under visible light are captured by the front camera, a tracking trajectory information variable is generated based on the package features and the corresponding package is tracked, and the human features are bound to the tracking trajectory information variable; the package features include appearance features and motion features; 2) When the X-ray camera captures the package features under X-ray, it matches the package features under X-ray with the package features under visible light in the generated tracking trajectory information variable, and updates the package features under X-ray to the successfully matched tracking trajectory information variable; 3) When new visible light package features and human features are captured by the rear camera, the new visible light package features are matched with the already generated tracking trajectory information variables, the new visible light package features are updated to the successfully matched tracking trajectory information variables, and the new human features are bound to the tracking trajectory information variables for the second time.

[0007] Furthermore, the tracking trajectory information variable includes the appearance features of the package at each moment before the current moment and the motion features of the previous moment; in step 1), tracking is performed in the following manner: when the appearance features in the package features collected at the current moment are consistent with the appearance features in the tracking trajectory information variable, and when the motion features at the current moment obtained based on the package features collected at the current moment are consistent with the motion features at the previous moment, the package features are updated to the tracking trajectory information variable.

[0008] Furthermore, in steps 2) and 3), matching is performed in the following manner: when the appearance features in the package features collected at the current moment match the appearance features in the tracking trajectory information variable, and the motion features at the current moment obtained based on the package features collected at the current moment match the motion features at the previous moment, the matching is successful.

[0009] Furthermore, the tracking trajectory information variable also includes the timestamp corresponding to the current frame, and the timestamp is used to calculate the time after the package enters the security inspection machine and to locate the time point when tracing back.

[0010] Furthermore, in step 1), if the tracking of the package fails, the appearance features in the package features corresponding to the package are compared with the appearance features in all the tracking trajectory information variables at that moment, and the package features that failed to be tracked are updated to the tracking trajectory information variable with the highest similarity.

[0011] Furthermore, after step 2), the tracking trajectory information variables that are not matched to the package features under the X-ray are corrected by the first error correction logic, and the first error correction logic is: The package features under X-ray detected by the X-ray camera in the successfully matched tracking trajectory information variable are recorded as X0, and the package features under visible light detected by the front camera are recorded as V1; for the tracking trajectory information variable that does not match the package features under X-ray but only contains the package features under visible light, the package features under visible light contained in this tracking trajectory information variable are recorded as V1 ＇ ; V1 ＇ Compare the similarity with X0 in the tracking trajectory information variable of the package features matched to the X-ray that have not passed the security inspection machine for a period of time, and select the one with the highest similarity to merge and obtain V1V1 ＇ X0; V1V1 ＇X0, connect X0 to V1 and V1 respectively ＇ Perform similarity comparison and remove V1 and V1 ＇ The features with lower similarity are used to obtain new tracking trajectory information variables that match the package features under X-ray.

[0012] Furthermore, after step 3), the tracking trajectory information variables that are not matched to the package features under the visible light of the rear camera are corrected by the second error correction logic, and the second error correction logic is: The package feature under visible light detected by the rear camera is recorded as V2, X0 in the tracking trajectory information variable containing only V1X0 is marked as X1, and X0 in the tracking trajectory information variable containing only X0V2 is marked as X2; Compare the similarity between V1 in V1X1 and V2 in X2V2, select the one with the highest similarity and merge them to obtain V1X1X2V2; in V1X1X2V2, calculate a Euclidean distance between X1 and X2 and V1 and V2 respectively, and retain only X1 and X2 with the smallest Euclidean distance to obtain tracking trajectory information variables that match the package features under X-ray, the package features under visible light of the front camera, and the package features under visible light of the rear camera.

[0013] Furthermore, after the second error correction logic, V1 and X1 in the tracking trajectory information variable that only contains V1X1 and has not been successfully matched within a period of time are respectively compared with V2 in the tracking trajectory information variable that only contains V2 and has not been successfully matched within a period of time. The one with the highest similarity is selected to merge to obtain V1X1V2. After the merger, it is considered that the corresponding package has been taken away; the X-ray package features in the tracking trajectory information variable that only contains the X-ray package features detected by the X-ray camera and has not been successfully matched within a period of time are recorded as X3, and X3 is compared with V2 in the tracking trajectory information variable that only contains V2 and has not been successfully matched within a period of time. The one with the highest similarity is selected to merge to obtain X3V2, and then X3 and V2 in the merged X3V2 are respectively compared with the one with the highest similarity. ＇ V1 in the tracking trajectory information variable ＇ Compare the similarities and select the one with the highest similarity to merge and get V1 ＇ X3V2, after merging, it is considered that the corresponding package has been taken away.

[0014] Furthermore, human body features include facial information, clothing information or body shape information.

[0015] A system for tracking and matching a package across modalities based on three-camera X-rays of the present invention comprises a processor, and the processor is used to execute instructions for implementing a method for tracking and matching a package across modalities based on three-camera X-rays.

[0016] The beneficial effects of the present invention are: The present invention is an improved invention. The present invention provides a method and system for cross-modal package tracking and matching based on three-camera X-ray, which tracks the package in real time across modalities and performs matching during the tracking process. The motion characteristics and appearance characteristics of the package are utilized during the matching process. Matching can also be performed when the package is delayed in entering or exiting the security inspection machine after being blocked by an obstacle, thereby improving the tracking and matching capabilities and image association capabilities of the package.

[0017] Furthermore, by comparing the similarity of relevant features for error correction, it is possible to solve the problem of difficulty in image association when the package is blocked by the security inspection equipment belt and cannot enter or exit the security inspection in time, or when the package overlaps and the tracking and matching are not successful. This can improve the human-package association and traceability capabilities of the security inspection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the security inspection process based on three-camera X-ray cross-modal package tracking and matching. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0020] The idea of ​​the present invention is to provide a method and system for cross-modal package tracking and matching based on three-camera X-ray, to track the package in real time across modalities, and to match it during the tracking process. The motion characteristics and appearance characteristics of the package are utilized in the matching process. In addition, errors are corrected by comparing the similarity of related features, thereby improving the person-package association and traceability capabilities of the security inspection system.

[0021] Method Example: like Figure 1 The security inspection process shown in the figure is based on three-camera X-ray cross-modal package tracking and matching. The security inspection device in the figure includes a front camera (Front camera), an X-ray camera (X-ray camera) and a rear camera (Rear camera). The security inspection process includes the following steps: 1) When a pedestrian carries a package into the Front camera area, the visible light package Front_bag and facial information are captured at multiple consecutive moments, the visible light package Front_bag is tracked, and the tracking information of the package is generated, and the facial information is bound to the tracking information of the package; 2) When the package enters the security inspection machine, the X-ray package information is captured, and the cross-modal matching network model is used to perform cross-modal matching between the X-ray package information and the Front_bag in the tracking information. If the match is successful, the X-ray package information is updated to the tracking information. If no match is found, the error correction logic is entered for error correction; 3) When the package comes out of the security inspection machine, it enters the Rear camera area and captures videos of the visible light package Rear_bag and facial information at multiple consecutive moments. The cross-modal matching network model is used to perform cross-modal matching between the Rear_bag and the X-ray package information in the tracking information. If the match is successful, the Rear_bag is updated to the tracking information. If no match is found, the error correction logic is entered for correction.

[0022] Specifically, the visible light package Front_bag tracking under the Front camera in step 1) specifically includes the following steps: 11) When a pedestrian carrying a package enters the Front camera area, the Front camera captures the face and visible light package Front_bag at multiple consecutive moments, extracts the corresponding package features through the cross-modal network matching model, extracts the face features through the neural network model, binds the face to Front_bag, and generates a tracking trajectory information variable Track_bag_id; Each Track_bag_id contains: (a) Appearance features of Front_bag at all times before the current moment extracted by the cross-modal network matching model (appearance features include shape, color, texture, etc.) (b) The motion feature of Front_bag at the previous moment (the motion feature is an eight-dimensional vector (x, y, a, h, vx, vy, va, vh), where (x, y) is the center point of the detection box (also called the bounding box), a is the aspect ratio of the bounding box, h is the height of the bounding box, and the last four dimensions are the speed or change rate of the first four variables, that is, the rate of change of the detection box in position, aspect ratio and height) (c) Timestamp of each moment in Front_bag (Timestamp is the specific time information of each moment, which is used to calculate the time interval in the subsequent steps and to trace the whole process and locate the time point of the problem when a problem occurs) 12) Perform detection-tracks comparison on multiple Front_bag detection information of the current frame and multiple Track_bag_ids to find the Track_bag_id that matches the motion features and appearance features in the Front_bag; 13) For the Front_bag whose trajectory is not tracked, the appearance features of the Front_bag are compared with the appearance features of the Track_bag_id of all existing locations before, and it is assigned to the Track_bag_id with the highest similarity.

[0023] Among them, in step 12), the specific steps of performing detection-tracks comparison on the detection information of multiple Front_bags of the current frame and multiple Track_bag_ids are as follows: 121) Input the existing multiple Track_bag_ids and multiple Front_bag detection information of the current frame (the multiple Front_bag detection information includes multiple Front_bag detection boxes (also bounding boxes, used to obtain corresponding motion features), multiple Front_bag appearance features and the timestamp at this moment); 122) Calculate the appearance cost matrix cost_matrix using the appearance features of multiple Front_bags detected in the current frame and the appearance features of multiple existing Track_bag_ids. The cost_matrix is ​​first obtained by calculating the distance (cosine distance) between the appearance features of the existing track and the appearance features detected at the current moment. The shape of cost_matrix is ​​[N1, N2], where N1 is the number of existing Track_bag_ids, and N2 is the number of targets of multiple Front_bags detected in the current frame. 123) Use the motion features of the previous moment to use Kalman filtering to predict the motion features of multiple existing Track_bag_ids; 124) Calculate the motion cost matrix motion_cost_matrix using the bounding box information of multiple Front_bags detected in the current frame and the motion features of the existing Track_bag_id. The shape of motion_cost_matrix is ​​also [N1, N2]. Then calculate the distance (Mahalanobis distance) between the Kalman filter prediction value of the current track and the detection result (i.e., the measurement value of the current frame). If the distance between a newly captured detection result and the prediction value exceeds the gating threshold of the Kalman filter, the corresponding cost matrix entry is set to a very large value, indicating that the match is unreasonable in terms of motion features. That is, the detection results that do not match the motion features are filtered out from the cost matrix through the gating threshold, and the cost_matrix in step 122) is updated using motion_cost_matrix. 125) Use cost_matrix to match the existing multiple Track_bag_ids and the multiple Front_bag information of the current frame. After the matching is completed, the Matches information is obtained. The Matches information contains the successfully matched tracks and detection results; 126) Update the track set. For multiple existing Track_bag_ids, update the motion features of the track and the appearance features of the Front_bag.

[0024] Among them, the basic principle of using Kalman filtering to predict motion features is: The Kalman filter updates the state estimate based on the state estimate at the previous moment and the current measurement value in each time step. Therefore, the Kalman filter is essentially dependent on time advancement. It predicts the state according to the passage of time and corrects the prediction when new measurement values ​​are obtained. However, when predicting, it not only depends on the time interval, but also considers the uncertainty of the state (covariance matrix), and uses the state estimate and covariance matrix of the previous moment to predict the state at the current moment. This management of uncertainty makes the Kalman filter more robust in the face of measurement noise and system model errors; after obtaining the measurement value at the current moment, it also uses these measurement values ​​to correct the predicted state, thereby obtaining a more accurate state estimate. This method is more flexible and accurate than simply relying on time interval matching. In addition, the Kalman filter uses a dynamic model to describe the process of system state changing over time. The dynamic model can consider the speed and acceleration of the movement, rather than just the relationship between position and time, which enables the Kalman filter to handle more complex and realistic motion situations.

[0025] Specifically, the specific steps for parcel matching under the X-ray camera in step 2) are as follows: 21) When the package enters the security inspection machine, X-ray imaging generates X-ray image information X-ray_bag, and the cross-modal matching network model extracts the X-ray package features; 22) Use the cross-modal matching network model to perform detection-tracks matching on the current multiple X-ray_bag detection information and multiple Track_bag_ids. For the matching steps, refer to steps 121) to 126. Replace the Front_bag with the X-ray_bag to obtain the Matches information. The Matches information contains the successfully matched tracks and detection results, and the track set is updated. For the existing multiple Track_bag_ids, the motion features of the tracks and the appearance features of the X-ray_bag are updated. 23) For the unmatched tracks unmatched_tracks, the error correction logic is performed, and the X-ray_bag feature detected by the X-ray camera is recorded as X0, and the feature of the visible light package Front_bag detected by the Front camera is recorded as V1; 24) For the trajectory that does not match the X-ray_bag feature but only contains the visible light package Front_bag feature, the visible light package Front_bag feature contained in this trajectory is recorded as V1 ＇ , the unmatched trajectory will be kept, and the visible light features saved in this trajectory will be compared with the system's temporary storage (the time when this trajectory stops updating t) after the package enters the security inspection machine Δt time ＇ to ＇ +Δt, Δt is calculated based on the timestamp in Track_bag_id, which is the time from each package entering the security inspection machine to leaving the security inspection machine) and all X-ray_bag features of the track before the right boundary of the camera X-ray (before leaving the security inspection machine) are matched for similarity, and the track with the highest similarity is selected for merging. The result V1V1 is obtained after merging. ＇ X0; 25) Targeting V1V1 ＇ X0, feature V1 and feature V1 ＇ Perform feature similarity matching with X-ray_bag respectively, remove the features with lower matching degree, and obtain the trajectory containing feature V1*X0, where V1* is V1 and V1 ＇ Features obtained after feature similarity screening.

[0026] Specifically, the visible light package Rear_bag matching under the Rear camera in step 3) specifically includes the following steps: 31) When the package comes out of the security inspection machine, the model recognizes the visible light package Rear_bag and facial information, and captures continuous videos of the visible light package Rear_bag and facial information. The cross-modal matching network model is used to extract the corresponding package features, and the facial features are extracted through the neural network model, and the facial features are bound to the corresponding package features; 32) Use the cross-modal matching network model to perform detection-tracks matching on the current multiple Rear_bag detection information and multiple Track_bag_ids. The matching steps refer to steps 121) to 126). Replace the Front_bag with the Rear_bag to obtain the Matches information. The Matches information contains the successfully matched tracks and detection results, and the track set is updated. For the existing multiple Track_bag_ids, the motion features of the tracks and the appearance features of the Rear_bag are updated. 33) Judge the matching result of the new detection (remember that the visible light feature of the current frame detection is V2, the X-ray features detected by the X-Ray camera are X1 and X2, and the visible light feature detected by the Front camera is V1). If the track matched by the current frame detection result only contains X-ray features (the features contained in this track in this step are recorded as X2V2), then V2 will be similar to the visible light features of all newer tracks (before the new detection position and within Δt time, Δt is calculated based on the timestamp in Track_bag_id) that do not contain V2. Judge, merge this trajectory with the trajectory with the highest similarity, and after merging, this trajectory contains the feature V1X1X2V2 (error correction process); if the trajectory matched by the current frame detection result contains both visible light features and X-ray features, no special processing is performed; for the trajectory containing the feature V1X1X2V2, X1 and X2 are respectively judged for similarity with all visible light features contained in this trajectory, and the one with the highest similarity is selected, and the X-ray feature with the second lowest similarity is eliminated. After elimination, the feature contained in the trajectory is V1X*V2, where X* is the feature obtained after X1 and X2 are screened by feature similarity; 34) Mark unmatched tracks (unmatched_tracks), which have no corresponding detection in the current frame and are temporarily missing; 35) Clean up the detection tracks that have not been paired for a long time, and compare the similarity between V1 and X1 in the tracking track information variables that only contain V1X1 and have not been successfully matched within a period of time (within 2 / 3Δt1, Δt1 refers to the running time from the crawler leaving the security inspection machine to the end of the crawler running) and V2 in the tracking track information variables that only contain V2 and have not been successfully matched within a period of time (V2 and V1 are judged by a similarity score of a, and V2 and X1 are judged by a similarity score of b, and the final score is (a+b) / 2). Select the one with the highest similarity (i.e. the highest score) and merge them to get V 1X1V2, after merging, it is considered that the corresponding package has been taken away; the X-ray package features in the tracking trajectory information variable that only contains the X-ray package features detected by the X-ray camera and has not been successfully matched within a period of time (within 2 / 3Δt2, Δt2 refers to the time from the leftmost end of the track to the time before entering the security inspection machine) are recorded as X3, and X3 is compared with V2 in the tracking trajectory information variable that has not been successfully matched within a period of time, and the one with the highest similarity is selected for merging to obtain X3V2, and then X3V2 is compared with V1 in the tracking trajectory information variable that has not been successfully matched within a period of time. ＇ V1 in the tracking trajectory information variable ＇ Perform similarity comparison (V1 ＇The similarity judgment score with V2 is a, and the similarity judgment score with X3 is b. The final score is (a+b) / 2). The one with the highest similarity (i.e. the highest score) is selected and merged to obtain V1. ＇ X3V2, after merging, the corresponding package is considered to have been taken away; in addition, for the trajectory that is directly matched successfully without error correction, after a period of time (2 / 3Δt1), the corresponding package is considered to have been taken away.

[0027] As another implementation, the facial information (facial features) in the above steps can be replaced by clothing information or body shape information or other human features that can distinguish pedestrians.

[0028] System Example: This embodiment provides a system for cross-modal package tracking and matching based on three-camera X-rays, which includes a front camera (Front camera) for capturing visible light package and facial information of a package before it enters a security inspection machine, an X-ray camera (X-ray camera) for capturing X-ray package information when the package enters the security inspection machine, and a rear camera (Rear camera) for capturing visible light package and facial information when the package comes out of the security inspection machine. The system also includes a processor for executing instructions for implementing a method for cross-modal package tracking and matching based on three-camera X-rays. The method for cross-modal package tracking and matching based on three-camera X-rays has been clearly introduced in the method embodiment and will not be repeated here.

Claims

1. A method for cross-modal package tracking and matching based on three-camera X-ray, characterized in that: The following steps are involved: 1) When the package features and human features under visible light are captured by the front camera, a tracking trajectory information variable is generated according to the package features and the corresponding package is tracked, and the human features are bound to the tracking trajectory information variable; the package features include appearance features and motion features; 2) When the X-ray camera captures the package features under X-ray, it matches the package features under X-ray with the package features under visible light in the generated tracking trajectory information variable, and updates the package features under X-ray to the successfully matched tracking trajectory information variable; 3) When new visible light package features and human features are captured by the rear camera, the new visible light package features are matched with the already generated tracking trajectory information variables, the new visible light package features are updated to the successfully matched tracking trajectory information variables, and the new human features are bound to the tracking trajectory information variables for the second time.

2. The method for cross-modal package tracking and matching based on three-camera X-ray according to claim 1 is characterized in that: The tracking trajectory information variable includes the appearance features of the package at each moment before the current moment and the motion features of the previous moment; in the step 1), tracking is performed in the following manner: when the appearance features in the package features collected at the current moment are consistent with the appearance features in the tracking trajectory information variable, and when the motion features at the current moment obtained based on the package features collected at the current moment are consistent with the motion features at the previous moment, the package features are updated to the tracking trajectory information variable.

3. The method for cross-modal package tracking and matching based on three-camera X-ray according to claim 2 is characterized in that: In the steps 2) and 3), matching is performed in the following manner: when the appearance features in the package features collected at the current moment match the appearance features in the tracking trajectory information variable, and the motion features at the current moment obtained based on the package features collected at the current moment match the motion features at the previous moment, the matching is successful.

4. The method for cross-modal package tracking and matching based on three-camera X-ray according to claim 1 is characterized in that: The tracking trajectory information variable also includes a timestamp corresponding to the current frame, and the timestamp is used to calculate the time after the package enters the security inspection machine and to locate the time point when tracing back.

5. The method for tracking and matching a package based on three-camera X-ray cross-modality according to claim 1, characterized in that: In the step 1), if the tracking of the package fails, the appearance features in the package features corresponding to the package are compared with the appearance features in all the tracking trajectory information variables at that moment, and the package features that failed to be tracked are updated to the tracking trajectory information variable with the highest similarity.

6. The method for cross-modal package tracking and matching based on three-camera X-ray according to claim 1, characterized in that: After step 2), the tracking trajectory information variables that are not matched to the package features under the X-ray are corrected by the first error correction logic, and the first error correction logic is: The package features under X-ray detected by the X-ray camera in the successfully matched tracking trajectory information variable are recorded as X0, and the package features under visible light detected by the front camera are recorded as V1; for the tracking trajectory information variable that does not match the package features under X-ray but only contains the package features under visible light, the package features under visible light contained in this tracking trajectory information variable are recorded as V1 ＇ ; V1 ＇ Compare the similarity with X0 in the tracking trajectory information variable of the package features matched to the X-ray that have not passed the security inspection machine for a period of time, and select the one with the highest similarity to merge and obtain V1V1 ＇ X0; V1V1 ＇ X0, connect X0 to V1 and V1 respectively ＇ Perform similarity comparison and remove V1 and V1 ＇ The features with lower similarity are used to obtain new tracking trajectory information variables that match the package features under X-ray.

7. The method for tracking and matching a package based on three-camera X-ray cross-modality according to claim 6, characterized in that: After step 3), the tracking trajectory information variables that are not matched to the package features under the visible light of the rear camera are corrected by the second error correction logic, and the second error correction logic is: The package feature under visible light detected by the rear camera is recorded as V2, X0 in the tracking trajectory information variable containing only V1X0 is marked as X1, and X0 in the tracking trajectory information variable containing only X0V2 is marked as X2; Compare the similarity between V1 in V1X1 and V2 in X2V2, select the one with the highest similarity and merge them to obtain V1X1X2V2; in V1X1X2V2, calculate a Euclidean distance between X1 and X2 and V1 and V2 respectively, and retain only X1 and X2 with the smallest Euclidean distance to obtain tracking trajectory information variables that match the package features under X-ray, the package features under visible light of the front camera, and the package features under visible light of the rear camera.

8. The method for cross-modal package tracking and matching based on three-camera X-ray according to claim 7, characterized in that: After the second error correction logic, V1 and X1 in the tracking trajectory information variable that only contains V1X1 and has not been successfully matched within a period of time are respectively compared with V2 in the tracking trajectory information variable that only contains V2 and has not been successfully matched within a period of time. The one with the highest similarity is selected to merge to obtain V1X1V2. After the merger, it is considered that the corresponding package has been taken away; the X-ray package features in the tracking trajectory information variable that only contains the X-ray package features detected by the X-ray camera and has not been successfully matched within a period of time are recorded as X3, and X3 is compared with V2 in the tracking trajectory information variable that only contains V2 and has not been successfully matched within a period of time. The one with the highest similarity is selected to merge to obtain X3V2, and then X3 and V2 in the merged X3V2 are respectively compared with the one with the highest similarity. ＇ V1 in the tracking trajectory information variable ＇ Compare the similarities and select the one with the highest similarity to merge and get V1 ＇ X3V2, after merging, it is considered that the corresponding package has been taken away.

9. The method for cross-modal package tracking and matching based on three-camera X-ray according to claim 1, characterized in that: The human features include facial information, clothing information or body shape information.

10. A system for tracking and matching packages across modalities based on three-camera X-rays, comprising a processor, characterized in that: The processor is used to execute instructions to implement the method for cross-modal package tracking and matching based on three-camera X-ray as described in any one of claims 1 to 9.

Images